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identified a number of criteria for the selection, development, and 
collection of network statistics and performance measures. Based on these 
criteria, and through extensive field-testing using a number of methods, the 
study team developed a core set of network statistics and performance 
measures. Key findings, issues, and recommendations include: (1) a core set 

of national network statistics and performance measures is possible; (2) a 
number of models exist to determine which statistics to use under what 
circumstances; (3) network statistics are evolutionary; (4) understanding, 
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EXECUTIVE SUMMARY 

This report summarizes the findings, issues, and lessons learned from the Developing 
National Public Library and Statewide Network Statistics and Performance Measures study 
conducted between January 1999 and August 2000 (IMLS Grant LL-80102). The overall goal of 
the study was to develop a core set of national statistics and performance measures that 
librarians, researchers, and policy makers can use to describe public library and library-based 
statewide network use of networked services and resources. More specifically, the study had the 
following key objectives: 

• Identify and develop a descriptive list of national statistics and performance measures 
that describe public library network use to include definitions, methods of collection, and 
intent/purpose of the elements and performance measures; 

• Test and validate the statistics in a sample of public libraries in selected states with the 
assistance of selected state library agencies and public libraries; and 

• Produce a concise manual that describes the proposed national statistics, possible 
performance measures that can be computed from these data elements and statistics, the 
techniques through which to collect the data, and considerations for analyzing and 
presenting network usage data. 

The study developed, tested, and refined the statistics and performance measures based on a 
number of field-testing and validating activities. 

The study focused on the development of network statistics and performance measures that 
would 1) aid public libraries assess the usage of their network services and resources and the 
local level for decision making, resource allocation, and other purposes; 2) provide scalable 
statistics and measures that generate statewide and national measures of public library network 
services resources usage; and 3) assist public librarians, state library staff, and policy makers 
understand the methods, techniques, and means through which to develop, collect, analyze, and 
present network usage statistics and performance measures through the development of a 
manual/guide. 

Study participants identified a number of criteria for the selection, development, and 
collection of network statistics and performance measures. These included: 

• Measures of Capacity, Use, or Impact. A traditional way of classifying library statistics 
is into input and output measures. A capacity measure describes the ability of a library to 
make use of a network resource or deliver a network service (e.g., the number of Internet 
workstations or the maximum speed of public access Internet workstations). A use 
measure describes the utilization of the library (e.g., the number of electronic reference 
transactions or visits to a library created WWW site). Impact describes the effects of 
library use (e.g., the number employed as a result of library network services or the 
number of newly literate readers as a result of library network services). 

• Measures that are Scalable. The collection of national network statistics and performance 
measures implies that it is possible to collect network statistics and performance measure 
data at the local library level and aggregate that data on a national and international basis. 



Bertot, McClure, & Ryan 



i 



November 16, 2000 



Developing Statistics and Performance Measures for the Networked Environment: Final Report 



For example, the data needs to be scalable in the sense that the data collected at the local 
library is important and useful for that library, for the state, and at a national level. 

• Measures that are Multipurpose. A national network measure will be used for a variety 
of purposes, some of which can be anticipated, at the range of levels mentioned above. 
These purposes include: seeking and determining library funding, planning for library 
services, managing resource allocation, shaping public opinion, educating the public, and 
developing policy, standards, sanctions and rewards. 

• Measures that can be Adaptable to Multi-Type Library Use. A national network measure 
gains in power if it represents all libraries rather than a special class. A key set of 
national network measures may well be supplied by licensed database vendors who sell 
most of their products to multi-type library consortia. Local and regional libraries of 
different types are increasingly partnering to share network resources and services. In all 
these cases, a common set of network measures is an asset. 

• Measures that Minimize the Potential for Misuse. Measures that must scale across 

governmental units, serve multiple purposes, and be useful across different types of 
libraries may have an increased potential for misuse or misinterpretation. 

• Measures that are Defined Clearly. The range of audiences for the proposed national 
measures suggests the need for measures that can be unambiguously defined, particularly 
with regard to their method of collection. Local librarians, in particular, asked for clear 
definitions that included examples where possible. 

• Measures that are Useful for Decision Making. Participants expressed a strong desire to 
avoid measuring something simply because one can. The danger of measuring because 
one can measure increases with the number and complexity of the criteria necessary to 
become a successful network measure. 

• Measures that Enable Comparison of Data Between Libraries. Peer comparisons with 
neighbors or with libraries that share some similar characteristics, within state or out of 
state are one of the most popular uses of the existing annual survey statistics. Such 
comparisons, however, can be a blessing or a curse, as they can lead to divisiveness and 
ill will and/or frustration when nothing was done or could be done to raise the standard of 
the lower library. 

• Impact of Network Measure on Compliance with Standards, Regulations, and Laws. All 
of the participating state library agencies use annual survey statistics to test compliance 
with various state standards (and their associated administrative regulations and laws) 
and in the determination of state aid to libraries. Network measures would be used as 
tests of compliance as well. Careful early attention is necessary to consider the likely 
outcomes of the use of any proposed network measure to test compliance. 

Based on these criteria, and through extensive field-testing using a number of methods, the study 
team developed a core set of network statistics and performance measures. 

Key Findings, Issues, and Recommendations 

Through the research process, the study discovered a number of key findings, issues, and 
recommendations for the development of national public library network statistics and 
performance measures. These include (the ensuing report presents considerable detail and depth 
on the selected items highlighted here): 
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• A core set of national network statistics and performance measures is possible. As 
identified in Figures 1 and 2 in the report, there are a number of what could be considered 
core network statistics in five key public library network service and resource areas - 
public access workstations, instruction, virtual visits, electronic services, and databases. 

• A number of models exist to determine which statistics to use under what circumstances . 
Depending on audiences to which the library and others are presenting, management 
needs, and other factors, the selection of network statistics and performance measures 
may differ. 

• Network statistics are evolutionary. Network statistics and performance measures, 
particularly those in this report, are not static and will need to evolve over time as 
technology changes and the implementation of technology in libraries changes. 

• Understanding, using, presenting network statistics and measures requires education. 
To incorporate network statistics and performance measures into library planning and 
evaluation activities will require librarian education in the understanding, methods, and 
analysis of the statistics. Likewise, library administrators, members of governance 
boards, and others will need education in the meaning and interpretation of such 
statistics. 

• Vendor community collaboration is key. Significant data — that of online database usage 
- is in the hands of the vendor community. It is essential that libraries and vendors work 
together to generate reports and reporting formats of interest to the library community. 
Similarly, it is important to consider a core set of database statistics that the vendor 
community, as a whole, can adopt and adhere to to minimize vendor reporting 
requirements. 

• Developing a national data collection system to move forward. It is critical to develop a 
national reporting system for public library network statistical data. The question is: 
what is that system and what should it look like? This question is the subject of a new 
IMLS-provided grant to the study team and will require resolution for there to be national 
statistics. 

The report details additional findings and provides details on those highlighted above. 

Next Steps 

The research into network statistics and performance measures continues: 

e The study team continues to maintain the project website initiated by this study 
(though now moved to a different location <http://www.ii.fsu.edu/ 
Albany_IMLS_Webfiles/public_html/>). 

• The study team reviews new technologies and the implication for those technologies 
in the development and presentation of network statistics and measures. 

• Individuals on the study team have involvement in a number of research projects 
reviewing, developing, and implementing network statistics and performance 
measures in a number of library settings. 

• Members of the study team continue to work with vendors and others working in the 
area of library network statistics and performance measures both in the U.S. and 
internationally. 
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• The study team is engaged in a new effort to develop a model for national public 
library data collection. 

These efforts will continue the process begun by this research project - the development of 
library network statistics and performance measures. Moreover, such efforts will facilitate 
across library definitions, methods, and reporting techniques as well as coordinate, where 
possible, efforts in this area by various groups (e.g., academic libraries, consortia, vendors, and 
publishers). 
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INTRODUCTION 

This report summarizes the findings, issues, and lessons learned from the Developing 
National Public Library and Statewide Network Statistics and Performance Measures study 
conducted between January 1999 and August 2000 (IMLS Grant LL-80102). This report has the 
following objectives: 

• Describe project research activities; 

• Summarize the findings from the study team’s various data collection activities; 

• Present the identified network statistics and performance measures; 

• Review project goals and objectives and the extent to which the study achieved those 
goals and objectives; and 

• Identify next steps and future directions for implementing a national model for collecting 
network statistics and performance measures in the public library community. 

This report, therefore, details project activities and results. 

Study Importance and Significance 

The development of national public library networked statistics and performance measures is 
an important undertaking that receives continued and increased attention and support. There is a 
broad recognition for the need of such statistics and performance measures that: 

• Assist libraries make a strong case for support for technology and information 
infrastructure by documenting their Internet-based services and resources; 

• Allow libraries to compare effectively themselves to others in terms of Internet 
development, costs, provision of services, connectivity, and use; 

• Enable library directors and administrative library agencies compete for resources with 
other organizations and/or departments by documenting the range, extent, and impact of 
library-provided networked services; 

• Facilitate the transition from traditional library use measures such as circulation, 
reference transactions, interlibrary loans, etc., to network measures that describe the 
nature and use of library-based network activities and resources; 

• Provide a decision-making framework for library staff, managers, and administrators to 
determine resource allocation strategies and meet other management needs; and 

• Provide a means through which to measure the quality of library services and resources in 
the networked environment. 

These and other factors point to the overall importance for research that generates public library 
network statistics and performance measures. 

This study focused on the development of network statistics and performance measures that 
would 1) aid public libraries assess the usage of their network services and resources and the 
local level for decision making, resource allocation, and other purposes; 2) provide scalable 
statistics and measures that generate statewide and national measures of public library network 
services resources usage; and 3) assist public librarians, state library staff, and policy makers 
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understand the methods, techniques, and means through which to develop, collect, analyze, and 
present network usage statistics and performance measures through the development of a 
manual/guide. 

Study Objectives 

The overall goal of the study was to develop a core set of national statistics and performance 
measures that librarians, researchers, and policy makers can use to describe public library and 
library-based statewide network use of networked services and resources. More specifically, the 
study had the following key objectives: 

• Identify and develop a descriptive list of national statistics and performance measures 
that describe public library network use to include definitions, methods of collection, and 
intent/purpose of the elements and performance measures; 

• Test and validate the statistics in a sample of public libraries in selected states with the 
assistance of selected state library agencies and public libraries; and 

• Produce a concise manual that describes the proposed national statistics, possible 
performance measures that can be computed from these data elements and statistics, the 
techniques through which to collect the data, and considerations for analyzing and 
presenting network usage data. 

The study therefore developed, tested, and refined the statistics and performance measures based 
on a number of field-testing and validating activities. Additional information about the project, 
including a study abstract and various study documents, can be found at 
<http://www.ii.fsu.edu/Albany_IMLS_Webfiles/public_html/>. 

Overview of Study Methodology 

The study used a multi-method approach to the development of national network statistics 
and performance measures. The data collection efforts encompassed a variety of data collection 
activities that involved library researchers, practitioners, policy makers, state library agencies, 
state library data coordinators, online database vendor representatives, and public library 
administrators and staff. In particular, the study team: 

• Worked with six (6) states throughout the data collection process (Delaware, Maryland, 
Michigan, North Carolina, Pennsylvania, and Utah). Within the states, the study team 
conducted site visits, focus groups, interviews, and surveys to assist in identifying, 
defining, and operationalizing the statistics and performance measures. Libraries within 
the states also served as the test-bed sites for testing and refining the initial statistics and 
performance measures. 

• Established a study advisory committee comprised of public librarians, state library 
officials, vendors, library professional organization officials (e.g., Urban Libraries 
Council, American Library Association), and policy makers to guide study research 
activities, identify network usage statistics and performance measures, pre-test data 
collection tools, and provide other advisory activities. 
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• Conducted focus groups, surveys, and interviews at ALA Midwinter 1999 and 2000, 
ALA Annual Conference in 1999 and 2000, the Public Library Association’s March 2000 
meeting, and the Federal State Cooperative System (FSCS) annual training workshop 
meetings in 1999 and 2000. Participants in the focus groups included public librarians, 
state library agency administrators and data coordinators, state librarians, vendors, policy 
makers, IMLS representatives, and others. The topics of the focus groups included 
desired data for decision making purposes, preliminary identification and definition of 
network statistics and performance measures, field-test implementation strategies, online 
database vendor data collection strategies, and the selection of final statistics and 
performance measures. 

• Conducted numerous interviews with state data coordinators, project principals of similar 
measurement efforts (e.g., Equinox in Europe, International Standards Organization 
efforts to incorporate network measures into library statistical collection efforts), and 
participants at a number of state and local library association, consortium, or related 
meetings. These interviews served to coordinate international network measurement 
activities to the extent possible, verify study research activities, gain a broader 
perspective on the types and nature of network statistics and measures most useful, and 
build consensus on the statistics and measures, their definitions, and the means through 
which to capture and report data. 

• Conducted focus groups, interviews, and presentations with a number of local and state 
library association meetings, FSCS working committees, and others upon the completion 
of the study to evaluate the study’s research process, products, and network statistics and 
performance measures. These activities also served to identify the next steps in 
implementing a national system of developing, collecting, and reporting national statistics 
and performance measures beyond the life of the project. 

Together, these various data collection efforts yielded a robust and multi-method approach to 
developing a core set of national network statistics and performance measures for public 
libraries. More detailed information regarding specific data collection instruments and tools is 
available in Appendix A. 



The Role of the States 



As indicated above, the study team worked extensively with six (6) states throughout the 
study. The selection of these states was based on the following criteria: 

• Willingness to participate in a multi-phase and longitudinal data collection effort; 

• Ability to provide the study team feedback on various study data collection instruments 
and efforts; 

• Ability to inform the study team of the issues identified in collecting network statistics 
and performance measures; 

• Diversity of network resources and services implementation within the state library and 
public libraries throughout the state; and 

• Willingness to spearhead network statistics and performance measure adoption efforts, 
first in their states, and then on the national level. 
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The states served as study advisors, participants, and testbeds. 



The ensuing sections present detailed study activities, findings, study evaluation activities, 
future directions, and supporting documentation. 

STUDY ACTIVITIES 

The study team engaged in a number of activities throughout the duration of the study to 
develop, test, and finalize the network statistics and performance measures in Figures 1 and 2. 
These activities included: 

• Project planning and advisory committee — October 1998- January 1999. The 
researchers assembled a study team during this phase of the project, mapped broadly 
project activities, established a project advisory committee (see Appendix B), identified 
the six states that would participate in the study, and developed a project web site. These 
activities culminated in the first meeting of the advisory committee at the 1999 ALA 
Midwinter meeting. 

• Preliminary identification of network statistics and measures - January 1999-March 
2000. During this phase of the project, the study team reviewed related literature and 
identified related research activities nationally as well as internationally. These efforts 
produced a number of possible statistics and performance measures to capture the use of 
public library networked resources and services. Appendix C shows the initially 
identified network statistics and performance measures. 

• Site visit planning and execution — April 1999-June 1999. The study team worked with 
liaisons in the six state library agencies to identify public libraries, consortia, vendors, 
and other relevant individuals/entities with whom to meet and/or coordinate site visit and 
data collection activities. Within the states, the study team met with state library 
personnel, public librarians, government officials, members of governing boards, vendor 
representatives, consortia managers, and others to gain an understanding of data 
collection needs, management decision making needs, and issues related to network 
statistics and performance measures. The study team also used the site visits to winnow a 
substantial list of possible statistics and measures to a core set for field-testing. 

• Field test planning and execution - July 1 999-December 1999. Working with the state 
library liaisons as well as liaisons within participating libraries, the study team developed 
a field-testing methodology for the statistics and performance measures. Each statistic 
was tested by at least two participating public libraries that differed in size, technology 
architecture, and metropolitan status (urban, rural, suburban) to account for variations in 
public libraries (see Figure 4). Each field-test liaison was sent a set of instructions that 
detailed the statistic(s), its definition, the process through which to collect data regarding 
the statistic, and the process through which to report the data. In addition, each field-test 
liaison completed a questionnaire regarding the collection process (e.g., burden), utility 
of the statistic, and any recommended modifications to the definition or collection 
process. Appendix D details the field-test procedures, methods, and considerations. 
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Figure 1. Public Library Network Statistics. 


Proposed National Statistic 


Definition 


Public Access Workstations 


# Public access workstations 


Annual count of the number of library owned public access 
graphical workstations that connect to the Internet for a dedicated 
purpose (to access an OPAC or specific database) or multiple- 
purposes. 


# Public access workstation users 


Annual count of the number of users of all of the library’s 
graphical public access workstations connected to the Internet 
computed from a one week sample. 


Maximum speed of public access 
Internet workstations 


Indication of the maximum bandwidth of public Internet access, 
e.g., less than 56kbps, 56kbps, 128kbps, 1 .5mbps, etc. 


Databases 


# Full text titles available by subscription 

Report: Serial titles, Other titles, Total titles 


Count of the number of full text titles that the library subscribes 
and offers to the public computed one time annually. 


# Database sessions 


Total count of the number of sessions (logins) initiated to the 
online databases. Definition adapted from proposed ICOLC 
standard http://www.library.yale.eduy consortia/webstats.html. 


# Database queries/searches 


Total count of the number of searches conducted in the library’s 
online databases. Subsequent activities by users (e.g., browsing, 
printing) are not considered part of the search process. Definition 
adapted from proposed ICOLC standard 
http://www.library.yale.edu/ consortia/webstats.html. 


# Items examined using subscription 
services 


Count the number views to each entire host to which the library 
subscribes. A view is defined as the number of full text 
articles/pages, abstracts, citations, and text only, text/graphics 
viewed. Definition adapted from proposed ICOLC standard 
http://www.library.yale.edu/ consortia/webstats.html. 


Electronic Services 


# Virtual reference transactions 


Annual count of the number of reference transaction using the 
Internet. A transaction must include a question received 
electronically (e.g., via e-mail, WWW form, etc.) and responded 
to electronically (e.g., e-mail). 


Public service time spent servicing 
information technology 

Report: Information technology staff, Paid public service 
staff (Professional Librarian, Paraprofessional), 
Volunteer, & Total 


Annual count of the staff hours spent in servicing information 
technology resource and service activity in public service areas 
computed based on a one week sample. 


Virtual Visits 


# Virtual visits to networked library 
resources 

Report: # Internal virtual visits, # External virtual 
visits, # Total virtual visits 


Count of visits to the library via the Internet. A visit occurs 
when an external user connects to a networked library resource for 
any length of time or purpose (regardless of the number of pages 
or elements viewed). Examples of a networked library resource 
include a library OPAC or a library web page. In the case of a 
user visit to a library web site a user who looks at 1 6 pages and 54 
graphic images registers one visit on the Web server. 


Instruction 


User information technology instruction 

Report: # Users instructed, # Hours of instruction 


A Count of the number of users instructed and the hours of 
instruction offered in the use of information technology or 
resources obtainable using information technology in structured, 
informal, and electronically delivered instruction sessions 
conducted or sponsored by the library. 


Staff information technology instruction 

Report: # Staff instructed, # Hours of staff instruction 


Annual count of the total number of staff instructed and the 
number of hours of formal instruction in the management or use of 
information technology or resources obtainable using information 
technology. 



Bertot, McClure, & Ryan 

ERIC 



5 November 16, 2000 

ji, 15 

BEST mpv mm ii ADI er 



Developing Statistics and Performance Measures for the Networked Environment: Final Report 



Composite/Performance Measure 


Definition 


Public access Internet workstation in proportion 
to the legal service area population 


The ratio of the legal service area population to # Public 
access Internet workstations . E.g., XYZ library provides 1 
public access Internet workstation per 3,000 legal service 
population. 


Average annual use per public access Internet 
workstation 


The ratio of the # public access Internet workstation users to 
the # public access Internet workstations. 


Total reference activity 


Combine traditional measures of reference service with 
electronic measures. 


% Virtual reference to total reference questions 


Percentage of # virtual reference transactions to total reference 
questions (both traditional and virtual). 


User information technology instruction as % of 
total reference activity 


The # users instructed in information technology as a percent 
of total reference activity. 


Level of paid public service effort in servicing 
information technology 


Percentage of paid public service staff time spent serving the 
public that is spent servicing information technology during a 
sample period. 


Total library materials use 


This composite measure combines the circulation and use 
figures for all of the paper, multimedia and electronic 
collections that the public library owns or provides access. 


% Electronic materials use of total library 
materials use 


Compares electronic materials use in the form of # Items 
examined using subscription services with the total library 
materials use. 


Total number of serial titles offered 


Count of paper based serials titles added to # Full text serial 
titles available by subscription 


% serial titles offered in electronic form 


Compares # Unique electronic full text serial titles available 
by subscription to the Total number of serial titles offered. 


Total library visits 


Physical attendance at the library and # virtual visits combined 
into one total. 


% Remote library visits 


The percent of virtual visits to total library visits (virtual plus 
physical library visits). 


% legal service area population receiving 
information technology instruction 


The percentage of the legal service area population receiving 
information technology instruction annually from the public 
library. 


Hours of formal information technology 
instruction per staff member 


The average number of hours of formal information 
technology instruction a public library staff member receives 
per year. 


# Users of electronic resources and services 


This composite figure adds # Virtual visits to networked 
library resources , # Users instructed in information technology 
and the # Virtual reference transactions. 
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Figure 3. Field Test Sites and Data Elements. 
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o Online vendor database data collection. A key component to the field test 
centered on online vendor database data. This portion of the study required a 
separate data collection effort as the participating libraries did not control directly 
their online database usage data - the vendors did. Appendix E details the 
process through which the study team engaged in the vendor database field test. 
The U.S. National Commission on Libraries and Information Science (NCLIS) 
played a key role in the study team’s work with the vendors, from hosting focus 
groups to working groups to assisting in the data collection process. 

• Finalize and validate the statistics and performance measures - January 2000-May 
2000. The field-test phase of the study yielded a variety of issues, concerns, and 
recommendations for the development of a final core set of national public library 
network statistics and performance measures. The study team engaged in a systematic 
review of the findings and developed a near-final set of statistics and performance 
measures, complete with definitions, data collection methods, and outstanding issues 
regarding the statistics and measures. The study team presented this list in a number of 
public forums to solicit feedback and guide the final statistics and measures selection 
process, to include definitions and methods of collection. Presentations and public 
forums occurred at the March FSCS training workshop, March Public Library 
Association conference, July American Library Association annual meeting, and a 
number of state and local library association meetings, panels, and workshops. 

• Produce network statistics and performance measurement manual - May 2000- 
October 2000. Based on the study research activities, research findings, data collection 
approaches, identified issues, and recommendations, the study team developed a concise 
manual to assist public librarians collect, analyze, and present the recommended network 
services and resources usage data. Copies of the manual, published by the American 
Library Association, accompany this report to IMLS (Bertot, J.C., McClure, C.R., and 
Ryan, J. (2001). Statistics and performance measures for public library networked 
services. Chicago, IL: American Library Association. ISBN: 0-8389-0796-2). 

• Evaluate the project process and products - March 2000-December 2000. The study 
team conducted a number of evaluation activities to assess the extent to which the process 
for developing national network statistics and performance measures was successful as 
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well as the study objectives were attained. These evaluation activities are discussed in 
more detail in the Study Evaluation section of the report. 

The number of research activities in which the study team engaged yielded a number of findings 
related to the development and implementation of national public library network statistics and 
performance measures. The following section of the report details selected study findings and 
issues. 



FINDINGS, ISSUES, AND CONSIDERATIONS 

Together, the study’s data collection efforts indicated that there are a number of models for 
the development of network statistics; a number of ways to define, collect, and use network 
statistics; a variety of data collection, managerial, and technical issues that require resolution for 
the development and collection of network statistics; and differing views and opinions as to the 
requirements, management, and coordination of national data collection efforts. 

Network Measures Selection Criteria 

Study participants identified a number of criteria for the selection, development, and 
collection of network statistics and performance measures. These include: 

• Measures of Capacity, Use, or Impact. A traditional way of classifying library statistics 
is into input and output measures. A capacity measure describes the ability of a library to 
make use of a network resource or deliver a network service (e.g., the number of Internet 
workstations or the maximum speed of public access Internet workstations). A use 
measure describes the utilization of the library (e.g., the number of electronic reference 
transactions or visits to a library created WWW site). Impact describes the effects of 
library use (e.g., the number employed as a result of library network services or the 
number of newly literate readers as a result of library network services). 

• Measures that are Scalable. The collection of national network statistics and performance 
measures implies that it is possible to collect network statistics and performance measure 
data at the local library level and aggregate that data on a national and international basis. 
For example, the data needs to be scalable in the sense that the data collected at the local 
library is important and useful for that library, for the state, and at a national level. 

• Measures that are Multipurpose. A national network measure will be used for a variety 
of purposes, some of which can be anticipated, at the range of levels mentioned above. 
These purposes include: seeking and determining library funding, planning for library 
services, managing resource allocation, shaping public opinion, educating the public, and 
developing policy, standards, sanctions and rewards. 

• Measures that can be Adaptable to Multi-Type Library Use. A national network measure 
gains in power if it represents all libraries rather than a special class. A key set of 
national network measures may well be supplied by licensed database vendors who sell 
most of their products to multi-type library consortia. Local and regional libraries of 
different types are increasingly partnering to share network resources and services. In all 
these cases, a common set of network measures is an asset. 
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• Measures that Minimize the Potential for Misuse. Measures that must scale across 

governmental units, serve multiple purposes, and be useful across different types of 
libraries may have an increased potential for misuse or misinterpretation. 

• Measures that are Defined Clearly. The range of audiences for the proposed national 
measures suggests the need for measures that can be unambiguously defined, particularly 
with regard to their method of collection. Local librarians, in particular, asked for clear 
definitions that included examples where possible. 

• Measures that are Useful for Decision Making. Participants expressed a strong desire to 
avoid measuring something simply because one can. The danger of measuring because 
one can measure increases with the number and complexity of the criteria necessary to 
become a successful network measure. 

• Measures that Enable Comparison of Data Between Libraries. Peer comparisons with 
neighbors or with libraries that share some similar characteristics, within state or out of 
state are one of the most popular uses of the existing annual survey statistics. Such 
comparisons, however, can be a blessing or a curse, as they can lead to divisiveness and 
ill will and/or frustration when nothing was done or could be done to raise the standard of 
the lower library. 

• Impact of Network Measure on Compliance with Standards, Regulations, and Laws. All 
of the participating state library agencies use annual survey statistics to test compliance 
with various state standards (and their associated administrative regulations and laws) 
and in the determination of state aid to libraries. Network measures would be used as 
tests of compliance as well. Careful early attention is necessary to consider the likely 
outcomes of the use of any proposed network measure to test compliance. 

Models for Developing Network Statistics and Performance Measures 

The study identified numerous models to use as a framework for developing statistics and 
performance measures (see Table 1 and Figure 2). Each model presents a different lens for 
developing statistics and performance measures. At this point, the study team is not indicating 
which model is the most appropriate - indeed, preliminary indications are that no one model is 
best; rather, each has its utility and can serve to provide a useful means for the development of 
statistics and performance measures. Moreover, there is a sense that it is possible to combine 
aspects of the models to generate statistics and measures. 

The Network Component Model 

As first described by Bertot and McClure (1998), this model provides a two-dimensional 
framework for the development of electronic statistics and performance measures (see Table 1). 
This model suggests that there are numerous components to electronic measures: 

• Technical infrastructure: The hardware, software, equipment, communication lines, 
and technical aspects of the network (e.g., workstations, modems, servers); 

• Information content: The information resources available on the network (e.g., local 
government information, special collections); 

• Information services: The activities in which users can engage and the services that 
users may use to complete various tasks (e.g., EbscoHost, UnCover, online applications); 
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• Support: The assistance and support services provided to help users better use the 
network (e.g., training, help desk); and 

• Management: The human resources, governance, planning, and fiscal aspects of the 
network (e.g., network staff, advisory boards, budgeting). 

In addition, there are different types of evaluation criteria that are possible to describe public 
library Internet-based use and services: 

• Extensiveness. How much of a service the network provides (e.g., number of users 
accessing a Web page per week, number of remote dial-ins per week); 

• Efficiency. The use of resources in providing or accessing networked information 
services (e.g., cost per session in providing access to remote users of an online database, 
or average number of times users are unable to successfully connect to the library’s 
servers); 

• Effectiveness. How well the networked information service met the objectives of the 
provider or the user (e.g., success rate of identifying and accessing the information 
needed by the user); 

• Service quality. How well a service or activity is done (e.g., percentage of transactions in 
which users acquire the information they need); 

• Impact. How a service made a difference in some other activity or situation (e.g., the 
degree to which network users enhanced their ability to gain employment or pursue 
business); 

• Usefulness. The degree to which the services are useful or appropriate for individual 
users (e.g., percentage of services of interest to different types of user audiences); and 

• Adoption. The extent to which institutions or users integrate and adopt electronic 
networked resources or services into organizational or individual activities (e.g., 
answering reference questions, generating inter-library loan requests). 

These types of criteria provide an important roadmap for thinking about the type of data 
elements and statistics that would be needed to produce such measures. 



Table 1. The Network Component Model. 
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The Audience Model 



In this approach, the consumer of the data is the primary lens for the development and 
collection of network statistics and performance measures (see Figure 4). Identified audiences 
include: 

• Library governance boards; 

• Local government officials (city, municipal, county); 

• State government officials (legislative and executive); 

• Fundors (primarily external funding entities such as granting institutions); 

• National policy making groups (e.g., Congress, Executive agencies); 

• Professional organizations (e.g., ALA, PLA); and 

• Researchers. 

Considerations for each of these audiences in terms of the statistics and measures are: 

• Informational needs; 

• Ability to interpret the data; 

• Utility of the data for decision making, policy formation, or other purposes; and 

• Comparability to other organizations under the purview of the audience members that 
engage in network-based activities (e.g., other city/county agencies that provide Web- 
based services). 

Together, the audience and the audience factors combine to form the basis for the development 
of network statistics and performance measures. 
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The Technology Infrastructure Model 



At present, public libraries and state library agencies have vastly differing levels of installed 
information technology infrastructure, level of services and content running over that 
infrastructure, and different plans for the development of library and state technology 
architectures. These differing levels of technology implementation have a direct impact on the 
relevance, feasibility to collect, and ability to use network statistics and performance measures. 
Thus, it is possible to develop statistics and performance based on a library’s or state library 
agency’s current state of technology infrastructure (see Figure 5). 

Based on discussions with study participants, reviews of the literature (see 
http://www.nysl.nysed.gov/libdev/edl/thirdpln.htm/, for example), and research conducted by the 
authors in previous studies (see, for example, Bertot and McClure, 1996; McClure, Bertot, and 
Beachboard, 1995), it is possible to identify stages of technology implementation at the library 
level. A rough means to categorize these levels are: 

• Level 1 — Stand-alone public access workstations that offer dial-up access to the Internet. 
Workstation may also provide access to CD-ROM-based material. May have access to 
online (Internet) databases through library subscription, but more likely through the state 
library’s licensing agreement; 

• Level 2 - Stand-alone public access workstations with dial-up connectivity to the 
Internet, plus a local area network that may also offer public Internet access through a 
dedicated leased-line connection. CD-ROM material access may be through network or 
single workstations. May have access to online (Internet) databases through library 
subscription as well as through the state library’s licensing agreement; 

• Level 3 - Networked facilities with leased-line access to the Internet. CD-ROM material 
access through network or single workstations. Access to online (Internet) databases 
through library subscription as well as through the state library’s licensing agreement. 
Library has a Web presence; and 

• Level 4 — Networked facilities with leased-line access to the Internet. CD-ROM material 
access through network or single workstations. Access to online (Internet) databases 
through library subscription as well as through the state library’s licensing agreement. 
Library has a Web presence and is developing unique content and resources for the site. 

Each level indicates a differing data collection and measurement need, level of effort required to 
collect the data, and an augmented level of understanding of technology and statistics to 
understand truly the meaning of the captured electronic data. The extent to which it is possible 
to develop a core set of network statistics and performance measures that are relevant, useful, 
and informative across all levels of technology infrastructure remain unclear at this time. 
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Figure 5. Technology Infrastructure Model. 




Composite Model 



While each of the above models - Network Component, Audience, and Technology 
Infrastructure - provides some notion of mutual exclusivity to the development, collection, 
maintenance, and interpretation of network statistics and performance measure, it is also possible 
to combine aspects of each model for the creation of network statistics and performance 
measures. 

Thus, public libraries and state library agencies may consider the audience of the intended 
statistics and performance measures while simultaneously considering the network components 
and evaluation criteria presented in the Network Component Model. Another approach might be 
to overlay the Technology Infrastructure Model on the Network Component Model, developing 
measures and statistics that account for the technology infrastructure of the library or state library 
agency for aspects (e.g., technology, technical support) of the network. 



The above models represent selected lenses for developing network statistics and 
performance measures identified by the study team thus far. There are likely other models that 
would provide libraries and state library agencies with a tool to develop network statistics and 
performance measures. 
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The study’s data collection activities identified several key conceptual, methodological, 
definitional, and other key issues regarding the development, implementation, management, and 
collection of national network statistics and performance measures. These issues are identified 
and discussed below. Table 2 summarizes the issues. 

Conceptual Issues 

Different Models/Frameworks for Developing National Network Statistics 

and Performance Measures 



There are a number of research models and frameworks for developing, defining, and 
measuring network statistics and performance measures. Each approach can affect the types of 
statistics and measures developed, the nature and use of those statistics and measures, the 
methodologies used to collect the statistic and measure data, and the presentation and 
interpretation of statistics and measure data. 

The way in which one frames or models approaches to develop a process for collecting and 
reporting national public library statistics and performance measures has a significant affect on 
which data to collect, how to organize the collecting process, and determining appropriate roles 
of key or lead state and federal agencies. For example, the study team currently is exploring a 
model that relies on Federal leadership, another on state leadership, and still another in which the 
statistics collecting and reporting process is “outsourced” to a non-governmental organization. 

Other models or approaches for studying this topic offer different means to fund such a 
national effort in a time when all government units (including local public libraries) appear to be 
extremely pressed to meet current commitments - to say nothing about supporting new 
responsibilities in the area of national data collection. These various models will need to be 
detailed and analyzed against a set of meaningful criteria to determine which of the approaches 
offers the best set of benefits to the various stakeholder groups participating in the process of 
developing this process. 

Core Statistics and Measures 

Is it possible to develop a limited number of core statistics and performance measures (6-8) 
that most states and libraries can agree upon for national data collection? Can this core set of 
statistics be part of a larger menu (12-15 additional statistics) from which libraries and states 
may select additional statistics and measures for collection and reporting? 

The findings from the site visits clearly confirm that local libraries do not have the time, 
expertise, or interest in committing significant resources to collecting data on a range of 
networked services and resources. The study team also found that proposed “core” networking 
statistics varied in usefulness from library to library and from state to state. Nonetheless, if the 
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Table 2. Issues in Developing National Statistics and Performance Measures for the 
Networked Environment. 


Conceptual Issues 


Issue 


Description 


Different Models/Frameworks for Developing 
National Network Statistics and Performance 
Measures 


There are a number of research models and frameworks for developing, defining, 
and measuring network statistics and performance measures. Each approach can 
affect the types of statistics and measures developed, the nature and use of those 
statistics and measures, the methodologies used to collect the statistic and measure 
data, and the presentation and interpretation of statistics and measure data. 


Core Statistics and Measures 


Is it possible to develop a limited number of core statistics and performance 
measures (6-8) that most states and libraries can agree upon for national data 
collection? Can this core set of statistics be part of a larger menu (12-15 
additional statistics) from which libraries and states may select additional statistics 
and measures for collection and reporting? 


Methodology Issues 


New and Variant Methodologies 


Network statistic and performance measure require researchers and professionals 
to consider the benefits and/or necessity of using traditional qualitative and 
quantitative methodologies (e.g., focus groups, interviews, surveys), adapting 
traditional methodologies (e.g., pop-up Web-based surveys), or creating new 
methodologies (e.g., Web-based transaction log analysis) to capture network 
usage data. 


Data beyond the Control of Libraries and States 


Libraries and states are engaging in substantial licensing agreements for Internet- 
based database access with vendors (e.g., OCLC, Ebsco, UM1, Gale/IAC). At 
present, libraries and states are dependent upon the vendors to provide them with 
database usage statistics. 


Samples rather than Populations 


To promote timely and responsive statistics and performance measures more 
reliance on carefully developed samples rather than 100% population responses 
may be needed. Such may be needed at the local, state, and national levels. 


Move to Qualitative Data 


To address a range of performance issues related to using networked information 
resources and services it may be necessary to increase the reliance on collecting, 
analyzing, and reporting various qualitative data. 


Estimates of Networked Services 


There are limited reliability and validity checks that can be established over the 
data collection process. There needs to be recognition that any of the statistics 
and performance measures likely to be proposed for the networked environment 
will also result in estimates and will have varying levels of accuracy depending on 
how they are collected and reported. 


Definitional Issues 


Defining Networked Services 


Currently using: 

Those electronic information resources and/or services that users access at a 
public, regional, or statewide library network. Examples of electronic network 
resources include: public, regional, or state library hosted or authored Web sites or 
library or licensed databases (e.g., Infotrac, SearchBank, EbscoHost). Examples 
of electronic network services include: provision of access to networks via public 
access workstation or dial in/remote access; network services such as email, 
listservs, chat, online reference/assistance; and training in the use of these 
resources and services. 

While there is agreement that networked statistics and performance measures 
must have clear and easily understandable definitions, many of these terms are 
complex and may require detailed definitions with detailed examples. Some of 
these definitions may have to include a bit of arbitrariness as there are competing 
possible definitions - none of which are compelling. 


Developing and Defining Network Statistics 
and Performance Measures 


Development of preliminary list of statistics/performance measures based on 
research/review of state library Web sites. 


Rethinking “Population Served:” 


In the networked environment, the “legal population served looses meaning since 
anyone from the around the word can access and obtain services off, for example, 
a library Website. Thus, traditional measures that index service provision or costs 
to legal population served (some local geographical area) may be inappropriate. It 
is unclear how such “per capita” measures can be translated into the networked 
environment. 
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Table 2. Issues in Developing National Statistics and Performance Measures for the 
Networked Environment (cont’d). 



Use of Networked Statistics and Performance Measure Issues 



Issue 


Description 


Evolving Development of Statistics and 
Performance Measures 


The process of developing network statistics and performance measures is 
evolutionary. As a result, the study team continues to refine network statistic and 
performance measure definitions, the statistics and measures of interest and use, 
and methodological issues regarding the collection and presentation of network 
statistics and measures based on the various data collection activities the study 
team undertakes. 


Implications for Longitudinal Aspects of 
Network Statistics and Performance Measures 


The rapidly changing nature of information technology will have a substantial 
impact on the life cycle of the network statistics and performance measures 
developed through this study (as well as those efforts undertaken by others 
working in this area). It is very likely that it will not be possible to have 
longitudinal data that extends beyond 3-5 years — at most. 


Coordinating the Development of Network 
Statistics and Performance Measures 


The importance of capturing library network services usage data is an issue with 
which numerous national and international library professional organizations, 
standards development committees, researchers, library organizations, library 
consortia and coalitions, and consultants have been wrestling substantially in 
recent times. These efforts yield, unfortunately, often uncoordinated approaches to 
the development of statistics and performance measures, differing statistic and 
measure definitions, variations on elements for data collection that reflect 
local/organizational issues and context, and the adoption of different data 
collection techniques. 


Comparability of Numbers Across Libraries 
and States 


Qualifications may be needed to explain the conditions and situations under which 
the data were collected, analyzed and reported as context for any type of 
comparability across libraries or across states. 


Reporting back to Libraries 


Until the states and national agencies do a better job of analyzing and reporting 
back to individual libraries statistics that are timely and useful for decision 
making, there will be limited commitment to collecting such data. Successful 
reporting back to individual libraries will require customized analyses, intended to 
assist them in local decision making, done in a very timely fashion. 



Library Technology and Uses Issues 



Technology Infrastructure and Configuration 


Based on the research conducted thus far by the study team, no two libraries have 
the same information technology infrastructure, configuration, or systems 
implementation. Moreover, while libraries may use similar applications and 
hardware, no two are implemented in the same manner. This creates a substantial 
challenge for the collection of the same data from libraries using similar (but 
different) technology in various configurations. 


My Technology, My Outlook 


A library facility’s infrastructure and use of technology leads to a local view of 
network statistics. That is, study participants base their need for electronic 
network statistics on their facility’s use of and involvement with network 
resources and services. As such, it is often difficult for individual libraries to see 
the need for certain statistics and performance measures that do not directly 
reflect their facility’s current implementation and use of various network services 
and resources. 



Data Collection Issues 



Automated Data Collection 


From both a Web-based environment as well as a vendor-supplied database 
environment, there is a need for networked services and resources to be described 
automatically and unobtrusively by the system itself rather than through overt data 
collection efforts on the part of state libraries and individual libraries. 


Working with Vendors 


Related to the issue above is the need for the library community to be more 
forceful in detailing contract language that requires database vendors to be better 
able to supply specific types of statistical information from the use of these 
databases. Increasingly, the library community will be dependent on statistics 
from vendors if such statistics are to be obtained. 
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Table 2. Issues in Developing National Statistics and Performance Measures for the 
Networked Environment (cont'd). 


Data Collection Issues (cont’d) 


Issue 


Description 


Statistics to Address Policy Issues 


The statistics and performance measures needed from year to year to address state 
and national policy issues are likely to vary. A national statistical system needs to 
be “fleet footed” enough to be able to both anticipate and respond timely to such 
policy issues affecting library services. 


Composite/Unit of Service Approach 


Many libraries and state library agencies are looking for some type of 
single/composite measure that captures the Units of Service that a library 
provides, rather than a single count/statistics that focuses on a single measure of 
library services (e.g., circulation). Such an approach removes the reliance upon 
single service measures such as circulation as indicators of library service 
consumption. The extent to which a composite measure that is valid, reliable, and 
useful is possible to create remains unclear at this time. 


Implementation Issues 


Developing Statistics and Measures Less the 
Problem 


There are numerous public library statistics and measures currently being 
collected by state library agencies to describe networked services, use, and 
resources. Concerns about comparability, the burden of such collection, accurate 
and timely national reporting, and implementing such a system nation-wide may 
be more problematic that developing the statistics and performance measures. 


Coordinating the Development of Network 
Statistics and Performance Measures 


The importance of capturing library network services usage data is an issue with 
which numerous national and international library professional organizations, 
standards development committees, researchers, library organizations, library 
consortia and coalitions, and consultants have been wrestling substantially in 
recent times. These efforts often cross-library types (e.g., academic, public), 
technology infrastructure and implementation, and other key factors. Each 
stakeholder realizes the importance of capturing electronic network usage data for 
a variety of reasons, and thus engages in efforts to develop network statistics and 
performance measures, definitions for those statistics and measures, and data 
collection techniques for those statistics and measures. 


Scalability of Network Statistics and 
Performance Measures 


The collection of national network statistics and performance measures implies 
that it is possible to collect network statistics and performance measure data atthe 
local library level and aggregate that data on a national basis. Preliminary data 
collection activities suggest that the scalability issue is complex and not easily 
resolved. 


Implementation of Procedures 


Any set of new networked statistics and performance measures will require a 
“roll-out” period that adequately educates and prepares libraries to collect the 
required data. This roll-out period will need to be coordinated by state library 
agencies and others, and it may take 1-2 years to complete prior to any actual data 
collection. 


Rewards and Incentives 


It is not clear if there are adequate rewards and incentives for state library and 
individual libraries to initiate a regime of collecting new networked statistics and 
performance measures given other demands on their time. There is some 
evidence that local libraries see the current system as completely unworkable, too 
time consuming, and providing useless data for them for local decision making. 


Ability to Collect Network Statistics and 
Performance Measure Data 


In order to attain national network statistics and performance measure data, it is 
necessary to collect the raw data at the local library outlet level. An issue raised 
in the study team’s data collection activities is the ability of the library outlets to 
collect such data. 


Leadership Issues 


Reliance on Key State Library Leaders 


State library agencies that are “leaders” in the collection of statistics and 
performance measures for networked information services and resources should 
be rewarded and encouraged to experiment with such efforts. Other states are 
likely to follow these efforts as they evolve. 


National Leadership 


There is a sense that the current national organization for selecting, collecting, 
analyzing, and reporting public library statistics may not be appropriate for the 
collection of national network statistics and performance measures. Should new 
models be considered for this task that include non-government agencies or other 
types of govemment/private section/library consortia arrangements? 
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Table 2. Issues in Developing National Statistics and Performance Measures for the 
Networked Environment (cont'd). 


Leadership Issues (cont 9 d) 


Issue 


Description 


Resources for the National Collection and 
Reporting Effort 


There is a need to consider the degree to which the U.S. government adequately 
supports the national data collection and reporting process for public libraries. An 
analysis of the costs associated with the current data collection program may be 
useful to determine which organizations are paying for what types of expenses 
related to the national data collection and reporting programs. 




Education Issues 


Re-Educating Local Community Leaders 


Librarians have spent decades convincing local governing boards that circulation 
counts or reference transactions, etc. that go up annually are a “good thing.” Now 
that these and traditional counts are stagnant or declining in many cases, librarians 
have to re-educate governing boards that web hits, electronic reference questions, 
full text down-loads, and other indicators are as or more important than the 
traditional measures. 


Education and Buy-in for Network Statistics 
and Performance Measures 


Although there appears to be substantial interest in the collection of network 
statistics and performance measures at the state library and national levels, there is 
mixed interest at the local library outlet and system level. 



nation is to have useful data to plan for and extend the role of public libraries in the networked 
environment, some agreement on which data are to be collected will be needed. 

Methodology Issues 



New and Variant Methodologies 



Network statistic and performance measure require researchers and professionals to consider 
the benefits and/or necessity of using traditional qualitative and quantitative methodologies (e.g., 
focus groups, interviews, surveys), adapting traditional methodologies (e.g., pop-up Web-based 
surveys), or creating new methodologies (e.g., Web-based transaction log analysis) to capture 
network usage data (see Table 3). 

Those collecting network-based statistics and performance measures, depending on the 
collection technique(s) selected, need to: 

• Know the capabilities and limitations of the selected methology(ies); 

• Ensure that like network aspects are being compared across often separate services (e.g., 
database usage from one vendor versus another); 

• Ensure the adherence to standard statistic definitions, operationalization, and collection 
techniques across libraries and states; 

• Understand precisely what the data represent in terms of network usage; and 

• Present the findings in a meaningful way to consumers of the data. 

The above represent substantial methodological challenges for the networked environment. 
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Table 3. Methodologies for Measuring Electronic Network Services. 


Qualitative 


Technique 


Function/Purpose 


Case Sites 


In-depth exploration of selected communities and target audiences in those communities, use of 
and involvement with the network. 

Use findings to inform broader quantitative data collection activities such as mail and electronic 
surveys. 


Content Analysis 


Gather various documentation and reports to review historical development and evolution of 
network-related activities. 


Focus Groups 


Explore identified key issue areas of network content, services, management, and performance. 

Use findings to inform broader quantitative data collection activities such as mail and electronic 
surveys. 


Small Group and 
Individual Interviews 


In-depth exploration of network content, services, management and performance with key 
project administrators and users. 

Assess the relationship between components of the network and future educational use and 
development of network resources. 

Use findings to inform broader quantitative data collection activities such as mail and electronic 
surveys. 


Critical Path Analysis 


In-depth exploration of user-based interactions with project-related components (e.g., training, 
workstation use, and searching). 

Use findings to uncover specific instance issues. Particularly appropriate for in-depth analysis 
of training and use issues. 


Quantitative 


Technique 


Function/Purpose 


Mail/Electronic 

Surveys 


Further explore identified key issue areas of network content, services, management, and 
performance with broader project population. 

Test findings from qualitative data collection activities with broader network population. 


Network Traffic 
Measures 


Collect network/terminal traffic use statistics such as users, user access points, information and 
service content use, and network server and router load. 

Provides sense of network load, capacity, and what services are used with what frequency. 


Web Log File Analysis 


Measure Web-based services by the analysis of Web server log files. 

Provides sense of users and locations from which access the services, server traffic, type of 
technology users have, and errors made. 
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Data beyond the Control of Libraries and States 

Libraries and states are engaging in substantial licensing agreements for Internet-based 
database access with vendors (e.g., OCLC, Ebsco, UMI, Gale/IAC). At present, libraries and 
state library agencies are dependent upon the vendors to provide them with a range of database 
usage statistics. Currently, the degree to which the database vendors respond to the library 
communities’ need for various statistics (e.g., log-ons to a particular database, IP addresses of 
log-ons, time of day of log-on, and duration of the session or visit to a particular database) is 
mixed at best. 

Based on a study in progress by the study team reviewing vendor database statistics and the 
site visits conducted, it is apparent that vendors provide reports to libraries and states that differ 
in their: 

• Format, time frame, presentation, and delivery (e.g., some send paper only reports 
monthly, some provide online/real time access); 

• Definitions of usage; and 

• Reporting, with some vendors providing a wealth of usage data (e.g., searches, sessions, 
time of day, logins, IP domains) and others providing only bare usage statistics (e.g., 
sessions and searches). 

Increasingly, however, libraries need to have summary statistics of networked database use that 
are controlled by vendors and not the library. This issue is not unique only to public libraries. 
All librarians will need to make known their data reporting needs to vendors better and agree 
among themselves as to the data needed; vendors will need to better listen and work with the 
library community to automate data collection and reporting in database software and 
standardize data reporting techniques. 

Samples Rather than Populations 

To promote timely and responsive statistics and performance measures more reliance on 
carefully developed samples rather than 100% population responses may be needed. Such may 
be needed at the local, state, and national levels. Currently, NCES strives to obtain 100% 
response rates before data will be analyzed and reported. The trade-off for obtaining 100% 
response from the population is an extended delay in reporting data. 

For many librarians and policymakers, having data based on samples that produces relatively 
quick reporting (six months from data collection to data reporting) is well worth a small decrease 
in the accuracy of the data. Part of the issue here is having data and findings that are “good 
enough” for decision making at the national, state and local level as opposed to data that have 
high academic quality for in-depth assessment. 
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Move to Qualitative Data 

To address a range of performance issues related to using networked information resources 
and services it may be necessary to increase the reliance on collecting, analyzing and reporting 
various qualitative data. Likert scales that assess user satisfaction with a particular networked 
service, for example, may be very useful to better describing perceived usefulness of a service 
such as interactive web-based reference service. The degree to which such approaches can then 
be compared across different libraries, however, is problematic. 

Additional research is necessary to assess the degree to which statistics and performance 
measures based on qualitative techniques are useful for various stakeholder groups. At the local 
level, it is well-known that anecdotal information and other types of “human interest stories” can 
be quite powerful in supporting the use of networked library services. At the state and national 
level, however, such may not be true and the anecdotes may be difficult to aggregate into a 
“national” perspective. Work, however, in this area should continue. 

Estimates of Network Services 



All nationally collected and reported data related to libraries and services are best seen as 
estimates - even those that are currently being collected. The fact of the matter is that there are 
limited reliability and validity checks that can be established over the data collection process. 
There needs to be recognition that any of the statistics and performance measures likely to be 
proposed for the networked environment will also result in estimates and will have varying levels 
of accuracy depending on how they are collected and reported. Estimates, however, are better 
than having nothing. Estimates can still be used as input for decision-making and are likely to be 
“good enough” as opposed to having no data. Providing footnotes and explanations to the 
limitations of such data should be considered as an important component in any data reporting 
process and can reduce the misuse of interpreting such data. 

Definitional Issues 



Defining Networked Services 



The definition that the study used for this term was: 

Network-based resources and services are those electronic information resources and/or 
services that users access at a public library or access via a public, regional, or statewide 
library telecommunications network. Examples of electronic network resources include: 
public, regional, or state library hosted or authored web sites or library or licensed 
databases (e.g., Infotrac, SearchBank, EbscoHost). Examples of electronic network 
services include: provision of access to networks via public access workstations or dial- 
in/remote access; network services such as e-mail, listservs, chat, online 
reference/assistance; and training in use of these resources and services. 

While there is agreement that networked statistics and performance measures must have clear 
and easily understandable definitions, many of these terms are complex and may require detailed 
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definitions with detailed examples. Some of these definitions may have to include arbitrary 
decisions as there are competing possible definitions - none of which may be compelling. Thus, 
in the short term it may be less important which definition of terms is used than that there is 
some national agreement to use a particular definition until there are compelling reasons to 
change it. 



Rethinking “Population Served” and “Per Capita” Measures 

In the networked environment, the “legal population served looses meaning because anyone 
from the around the word can access and obtain services off, for example, a library Website. 
Thus, traditional measures that index service provision or costs to legal population served (or 
some local geographical area) may be inappropriate. As an example, the performance measure 
“reference transactions per capita” can include electronic reference transactions that might 
originate from around the world. But in fact the determination of the “per capita” part of the 
measures is based on an estimate of legal population served - which may come from the state or 
another government unit. It is unclear how such “per capita” measures can be translated into the 
networked environment. 

Issues Related to the Use of Networked Statistics and Performance Measures 

Evolving Development of Statistics and Performance Measures 

The process of developing network statistics and performance measures is evolutionary. As 
a result, the study team continues to refine network statistic and performance measure 
definitions, the statistics and measures of interest and use, and methodological issues regarding 
the collection and presentation of network statistics and measures based on the various data 
collection activities the study team undertakes. There may be a period of time where these 
definitions change over time because the ability to measure networked services and resources 
may also change over time. Thus, definitions that make sense today, may be more reliable or 
accurate once research methods and measurement techniques improve. 

More importantly, however, is that there is a need to revisit the core statistics and measures, 
definitions, and data collection techniques periodically. Technology infrastructure and 
capabilities change rapidly and, as a result, there will be an impact on the types of statistics and 
measures of interest, ability to collect those statistics and measures, and the presentation and 
interpretation of those statistics and measures. 
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Implications for Longitudinal Aspects of Network Statistics and Performance Measures 

The rapidly changing nature of information technology will have a substantial impact on the 
life cycle of the network statistics and performance measures developed through this study (as 
well as those efforts undertaken by others working in this area). It is very likely that it will not 
be possible to have longitudinal data describing networked information resources and services 
that extend beyond 3-5 years. The mindset of producing statistics that are meaningful over long 
periods of time (e.g., circulation per capita; attendance counts; reference transactions; etc.) may 
need to change. 

For example, collecting data to describe the degree to which public libraries in the US are 
connected to the Internet was especially important in the early to mid-1990s. However, as that 
percentage approaches 90% (as it is expected to do early in the new century, its importance is 
considerable less. More important might be describe the type of connectivity and the types of 
services being provided by that connection. Five years from now, it may be less important to 
describe the type of public library connectivity if 90% or more have T-3 or better levels of 
connectivity. By then, some new type of statistic - not previously considered because of 
changes in technology and access — may be more appropriate to describe library networked 
services and resources. 

The result is that the library community will need to adopt a different model for data 
collection activities. This model is one of: 

• Rapid response and focused data collection activities; 

• Multiple data collection techniques, including online and print surveys, interviews, focus 
groups, logs, etc.; 

• Quick analysis and presentation of results; and 

• Continually evolving data elements and measures. 

Such a model requires a new philosophical approach by the library profession towards the 
collection of long-term library statistics as well as the development of new data collection 
mechanisms between libraries and the state library agency responsible for compiling and passing 
on library statistics to national data collection entities. 

Comparability of Numbers Across Libraries and States 

To what extent can these national statistics and performance measures be compared across 
libraries and states? The answer lies, at least in part, on having carefully developed procedures 
and definitions that facilitate libraries collecting data the same way and under similar conditions. 
The issue of comparability and accuracy of the data across different states and libraries is not a 
new issue. The results of the site visits, however, suggest that insuring accuracy or data so that 
such comparability can occur may be more challenging for networked services and resources 
than for traditional services currently being collected. 
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Reporting back to Libraries 

Until the states and national agencies do a better job of analyzing and reporting back to 
individual libraries statistics that are timely and useful for decision-making, there will be limited 
commitment to collecting such data. Successful reporting back to individual libraries will require 
customized analyses, intended to assist them in local decision making, done in a very timely 
fashion. 

A common complaint heard by the study team during the site visits was that much of the data 
they collected was not reported back to them in a manner in which they could use it for local 
decision making. Some interesting approaches, however, are evolving. Some states, such as 
Kentucky <http://www.kdla.state.ky.us/libserv/stats.htm>, are currently experimenting with 
interactive websites that allow local libraries to analyze the state database of statistics and print 
off their own reports and graphs. Some private sector firms such as Bibliostat 
<http://www.bibliostat.com/> are implementing systems that allow libraries to do real time 
comparisons and benchmarking against other “peer libraries.” Additional work needs to be done 
in this area to promote ways in which local libraries can make better use of the data they collect. 

Issues Related to Library Technology and Uses 

Technology Infrastructure and Configuration 

Based on the research conducted thus far by the study team, no two libraries have the same 
information technology infrastructure, configuration, or systems implementation. Moreover, 
while libraries may use similar applications and hardware, no two are implemented in the same 
manner. This creates a substantial challenge for the collection of the same data from libraries 
using similar (but different) technology in various configurations. Another implication is that the 
development for instructions to libraries about how best to collect networked data may require 
multiple approaches to accommodate differences in local technology infrastructure 
configuration. 

At issue is not the development of and adherence to the definitions of the electronic statistics 
and performance measures. Rather, the key issue is being able to collect, given the differing 
configurations and implementations, the same data across the various implementations and 
technologies. It may be that libraries and state library agencies will need to undergo modification 
to be able to generate and capture some of the network statistics and performance measures 
recommended by this research effort. 

My Technology, My Outlook 

A library facility’s infrastructure and use of technology leads to a local view of network 
statistics. That is, study participants base their need for electronic network statistics on their 
facility’s use of and involvement with network resources and services. As such, it is often 
difficult for individual libraries to see the need for certain statistics and performance measures 
that do not directly reflect their facility’s current implementation and use of various network 
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services and resources. Generally, librarians want to collect networked data only if it makes 
sense for their particular situation and for their particular technology. 

Data Collection Issues 



Automated Data Collection 



From both a Web-based environment as well as a vendor-supplied database environment, 
there is a need for networked services and resources to be described automatically and 
unobtrusively by the system itself rather than through overt data collection efforts on the part of 
state libraries and individual libraries. For example, there are a number of software programs 
that can track uses and services provided via the library’s website. Libraries may also be able to 
write their own programs to track/monitor web and other types of uses. Many librarians 
commented that they simply do not have the time, expertise, or interest in collecting such data 
for the state or the national government unless the data collection process is automated and 
unobtrusive. 



Working with Vendors 

Related to the issue above is the need for the library community to be more forceful in 
detailing contract language that requires database vendors to be better able to supply specific 
types of statistical information from the use of these databases. Increasingly, the library 
community is dependent on statistics from vendors if such statistics are to be obtained. Progress 
could be made on this issue if lead libraries would agree amongst themselves as to the type and 
definition of the various statistics that should be provided by the vendors. The database vendors 
cannot be expected to provide customized and/or unique sets of statistics to each of their 
customers. 

A complicating factor in working with vendors is that there are a number of initiatives that 
claim library representation with online database vendors. For example, the International 
Consortium of Library Consortia (ICOLC), Association of Research Libraries (ARL), Council 
on Library and Information Resources (CLIR), U.S. National Commission on Libraries and 
Information Science (NCLIS), to name a few, are working towards developing vendor statistics 
and reporting standards. Working with and coordinating among these efforts became a focus of 
this study. The extent to which continued collaboration among these efforts will be possible in 
the future is unclear. 



Composite/Unit of Service Approach 

Many libraries and state library agencies are looking for some type of single/composite 
measure that captures the Units of Service that a library provides, rather than a single 
count/statistics that focuses on a single measure of library services (e.g., circulation). Such an 
approach removes the reliance upon single service measures such as circulation as indicators of 
library service consumption. For example, some librarians recommended to the study team that a 
unit of service might be “networked services provided” which would include electronic reference 
requests, log-ins on the library’s website, and downloads of full text articles from the library (or 
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database provider’s) server. The extent to which a composite measure that is valid, reliable, and 
useful is possible to create remains unclear at this time. 

Statistics to Address Policy Issues 



The statistics and performance measures needed from year to year to address state and 
national policy issues are likely to vary. A national statistical system needs to be “fleet footed” 
enough to be able to both anticipate and respond timely to such policy issues affecting library 
services. For example, a current debate in the U.S. is the role of libraries in supporting the 
development of universal service to the nation’s residents. When the initial debates occurred in 
Congress regarding universal service issues (as described in the Telecommunications Act of 
1996), the library community had very little data to address these issues and propose strategies 
for appropriate roles of public libraries to support the provisions of universal service in the new 
law. 



The Burden of Data Collection at the Local Library 



To reduce overall burdens of data collection in may be appropriate to consider collecting 
certain data to produce certain statistics and performance measures every other year or every 
third year, etc. In some instances a serious reduction in data currently being collected may be 
necessary if new networked statistics are to be collected. Many local public librarians report that 
the existing annual surveys administered by the state library agencies are “excessive” in time 
demands. Indeed, some state library agencies add hundreds of additional data elements to the 
annual survey beyond those required by the NCES. 

Implementation Issues 

Developing Statistics and Measures Less the Problem 

The study team has identified numerous public library statistics and measures currently being 
collected by state library agencies that describe networked services, programs, use, and resources 
<http://www.ii.fsu.edu/Albany_IMLS_Webfiles/public_html/index.html>. The study confirmed 
that some states are also developing additional statistics and performance measures for statewide 
collection. 

Concerns about data comparability, the burden of such collection efforts, ensuring accurate 
and timely national reporting, and implementing such a system nation-wide may be more 
problematic that developing and agreeing on the statistics and performance measures to be used 
as a basis for national data collection and reporting. A major issue encountered by the study 
team during the site visits was how best to reduce the potential list of statistics related to 
networked services and resources. 

Coordinating the Development of Network Statistics and Performance Measures 

The importance of capturing library network services usage data is an issue with which 
numerous national and international library professional organizations, standards development 
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committees, researchers, library organizations, library consortia and coalitions, and consultants 
have been wrestling substantially in recent times. These efforts often cross-library types (e.g., 
academic, public), technology infrastructure and implementation, and other key factors. Each 
stakeholder realizes the importance of capturing electronic network usage data for a variety of 
reasons, and thus engages in efforts to develop network statistics and performance measures, 
definitions for those statistics and measures, and data collection techniques for those statistics 
and measures. 

These efforts yield, unfortunately, often uncoordinated approaches to the development of 
statistics and performance measures, differing statistic and measure definitions, variations on 
elements for data collection that reflect local/organizational issues and context, and the adoption 
of different data collection techniques. 

The study team continues to identify existing electronic network statistic and performance 
measurement efforts, and is attempting to coordinate efforts where possible for the purpose of 
developing standard statistics and performance measure definitions and data collection 
techniques. The extent to which these efforts can inform the development of national public 
library and statewide network statistics and performance measures remains unclear. 

Scalability of Network Statistics and Performance Measures 

The collection of national network statistics and performance measures implies that it is 
possible to collect network statistics and performance measure data at the local library level and 
aggregate that data to a state and national basis (see Figure 6). Preliminary data collection 
activities suggest that the scalability issue is complex and not easily resolved. 



Figure 6. Levels of Collection and Aggregation for Statistics and Performance 
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For example, the data needs to be scalable in the sense that the data collected at the local 
library is important and useful for that library, for the state, and at a national level. If the local 
library believes that the data is not useful locally, librarians may not collect the data nor be 
committed to collecting it accurately. Scalable also implies that when the data from all the 
various local libraries and the 50 states are collated or aggregated, they are still useful and 
meaningful. 



Implementation of Procedures 

Any set of network statistics and performance measures will require a “roll-out” period that 
adequately educates and prepares libraries to collect the required data. This roll-out period will 
need to be coordinated by state library agencies and others, and it may take 1-2 years to complete 
prior to any actual data collection. 

It has been suggested to the study team during the site visits that that any new procedures for 
data collection (including the introduction of new data elements) should first be pre-tested with 
collection instructions. After the pre-test, the revisions and changes should be confirmed as 
appropriate with selected representatives at state library agencies. Then the FSCS coordinators 
at the individual states should be briefed and trained. Finally, the local libraries would be briefed 
on the data collection process and be given a year to prepare. 

Rewards and Incentives 



It is not clear if there are adequate rewards and incentives for state library and individual 
libraries to initiate a regime of collecting new networked statistics and performance measures 
given other demands on their time. There is some evidence that local libraries see the current 
system as completely unworkable, too time consuming, and providing useless data for them for 
local decision making. 

A key aspect of successfully implementing a national program to collect and report statistics 
describing networked activities and services will be also implementing a reward and incentive 
system for local libraries to want to contribute to the program. Additional thought will be 
necessary to identify a range of possible incentives and rewards that encourage local libraries to 
participate. The assumption that local libraries will continue to participate in the annual surveys 
that includes additional data collecting requirements related to networked services and resources 
without such incentives should be re-examined. 

Ability to Collect Network Statistics and Performance Measure Data 

In order to attain national network statistics and performance measure data, it is necessary to 
collect the raw data at the local library outlet level. An issue raised in the study team’s data 
collection activities is the ability of the library outlets to collect such data. Indeed, branch library 
managers and library system directors expressed concern in the ability of their facilities to: 

• Engage in rigorous data collection activities that require staff time and effort; 

• Support data collection activities with staff and resources; 
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• Develop expertise in electronic network data collection activities; and 

• Collect any other data than that generated by system activity logs. 

Essentially, the primary source of data collection - the library outlet level - may not have the 
ability, resources, or expertise to engage in electronic network data collection activities. 

Leadership Issues 



Reliance on Key State Library Leaders 



State library agencies that are “leaders” in the collection of statistics and performance 
measures for networked information services and resources should be rewarded and encouraged 
to experiment with such efforts. A number of states are already experimenting with the collection 
of such data. Some are using innovative techniques for collecting statewide data electronically. 
Still others are developing websites that allow for interactive use and analysis of the annual 
survey data. Other states, once made aware of these innovations, are likely to follow these 
efforts as they evolve. 



National Leadership 



There is a sense that the current national organization for selecting, collecting, analyzing, and 
reporting public library statistics may not be appropriate for the collection of national network 
statistics and performance measures. As mentioned earlier, the process for recommending and 
then implementing the collection of new data elements is cumbersome and time-consuming at 
best. The time lag between submission of data and the reporting of that data is excessive - 
sometimes two years or more. 

The key organizations involved in this national data collection and reporting process 
currently are the U.S. National Center for Educational Statistics, the U.S. National Commission 
on Libraries and Information Science (NCLIS), the Institute on Museums and Library Services 
(IMLS), the state library agencies. New models and organizations (as well as the possible 
involvement of those that participate currently in the collection of library statistics) should be 
considered for this program that include non-government agencies, educational institutions, or 
other types of government, private sector/library consortia arrangements. 

Resources for the National Data Collection and Reporting Effort 

Another factor to consider is the degree to which the U.S. government adequately supports 
the national data collection and reporting process for public libraries. While it certainly is true 
that government agencies everywhere are finding budgets to be tight, an analysis of the costs 
associated with the current data collection program may be useful to determine what 
organizations are paying for what types of expenses related to the national data collection and 
reporting program. 

Further, some thought should be given to any additional expenses that might be incurred by 
various organizations if the data collection and reporting process is to be expanded with the 
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addition of a number of new data elements related to networked services and resources. It may 
be that some approaches for expanding the program to cover networked services and resources 
may be more cost-effective than others. 

Education Issues 



Re-Educating Local Community Leaders 

Librarians have spent decades convincing local governing boards that circulation counts, 
attendance records, reference transactions, etc. that go up annually are a “good thing.” Now that 
these and other traditional counts are stagnant or declining in many cases, librarians have to re- 
educate governing boards that web hits, electronic reference questions, full text down-loads, and 
other indicators are as or more important than the traditional measures. 

The move to delivery of and access to a range of electronic services and resources via the 
networked environment by libraries in recent years has not been “counted” or adequately 
reported as part of the overall presentation of public library activities. Thus, to some extent, 
public libraries are inadequately representing and describing what it is they do in and for the 
local community. Local community leaders may not understand services delivery and use in a 
networked environment. In such cases, re-education will be more than explaining the use of 
different or augmented statistics to describe library services in this evolving networked 
environment. 

Education and Buy-in for Network Statistics and Performance Measures 

Although there appears to be substantial interest in the collection of network statistics and 
performance measures at the state library and national levels, there is mixed interest at the local 
library outlet and system level. Some librarians have yet to recognize the impact and 
implications arising from the move to delivery of information services in a networked 
environment - nor, the changing role of the library in this environment. There also is some 
limited appreciation of the usefulness and importance of having national data related to the 
public library in the networked environment to propose and debate various policy issues in 
Congress and the administration. 
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EVALUATING THE PROJECT AND PROJECT RESULTS 

This section evaluates the extent to which the research study accomplished its goals and 
objectives. The study team engaged in a number of formative and summative evaluation 
processes through which to assess and validate the study’s activities, findings, and 
recommendations. 

Study Objectives 

The overall goal of the study was to develop a core set of national statistics and performance 
measures that librarians, researchers, and policy makers can use to describe public library and 
library-based statewide network use of networked services and resources. The study had the 
following three objectives: 

1. Identify and develop a descriptive list of national statistics and performance measures 
that describe public library network use to include definitions, methods of collection, and 
intent/purpose of the elements and performance measures; 

2. Test and validate the statistics in a sample of public libraries in selected states with the 
assistance of selected state library agencies and public libraries; and 

3. Produce a concise manual that describes the proposed national statistics, possible 
performance measures that can be computed from these data elements and statistics, the 
techniques through which to collect the data, and considerations for analyzing and 
presenting network usage data. 

As such, the study set out to identify national network statistics and performance measures for 
public libraries, test and validate those statistics and performance measures, and produce a 
manual that described the statistics and measures and the means through which to collect data on 
those statistics and measures. 

Identify and Develop a List of National Statistics and Performance Measures 

The study team engaged in a number of activities to identify potential network statistics and 
performance measures, including (see Appendix C): 

• Literature review of network statistical and performance measurement activities in the 
library and other literature; 

• Focus groups and interviews with state library agency staff (including state data 
coordinators), public librarians, library association staff (e.g., Urban Libraries Council, 
Public Library Association), policy makers, library governing board members, and 
others; and 

• Surveys with state library agency staff (including state data coordinators), public 
librarians, library association staff (e.g., Urban Libraries Council, Public Library 
Association), policy makers, library governing board members, and others regarding the 
utility, interest, and ability to collect suggested statistics and.performance measures. 
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This extensive review and pre-testing process yielded a core set of statistics and performance 
measures, definitions, data collection methodologies, and reporting procedures for field-testing 
purposes. 



Test and Validate Statistics 



As described in the Methodology and Study Activities sections of this report, as well as 
Appendices D and E, the study team engaged in extensive testing and validation activities 
regarding the network statistics and performance measures. To summarize, the study team: 

• Field-tested all statistics in at least two public library settings over a period of six months. 
During this time, field-test sites not only attempted to collect the statistics using the 
prescribed methodologies and techniques, but also provided suggestions and guidance as 
to modifications in the definition, methods of collection, and analysis and presentation of 
the statistics. 

• Conducted a separate field-test of the online vendor database statistics using individual 
public libraries, a multi-type library consortia (North Carolina’s NCLive), and 
Maryland’s statewide network Sailor. Through these entities, the study team worked 
with a number of vendors to collect and report usage statistics for a one-month period for 
each of the entities. 

• Validated the final statistics and performance measures through presentations, 
workshops, focus groups, interviews, and other activities upon completion of the field- 
test. Examples of these activities include: 

o Presentation at the FSCS data coordinators’ annual training workshop in March 

2000, 

o Coordination with the FSCS working group on statistics, and NCLIS, to begin 
the adoption process for selected network statistics into the annual public library 
collection process (still ongoing), 

o Presentations and focus groups with Maryland public library administrators 
(MAPLA) and directors at their spring and fall 2000 meetings, 

o Presentations and focus groups with the Illinois library directors at their annual 
meeting in August 2000, 

o Focus groups and interviews with librarians, vendors, policy makers, and others 
at ALA Midwinter 2000, the PLA conference in March 2000, and ALA Annual 
conference in July 2000, 

o Focus groups, interviews, and presentations with librarians and managers at the 
International Information Resources Management Association conference in May 
2000, 

o Presentation and feedback from librarians at a regional library system in 
Wisconsin, September 2000, 

o Presentation, small group meetings, and feedback from librarians at the State 
Library of Florida annual Director’s Conference, October 2000, 

o Presentation and dissemination at the Panhandle Library Access Network, 
Panama City, FL, October 2000, 

o Presentations, feedback, and dissemination to librarians and state library officials 
at the Pennsylvania Technology Conference, November 2000. 
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Together, these activities served to produce valid and reliable network statistics and performance 
measures. To the degree that these presentations and meetings occurred later in the project, they 
also contributed toward dissemination of project findings and results. 

Network Statistics and Performance Manual 



Once the study team produced a final set of tested and validated network statistics, the team 
focused on developing a manual to guide public librarians through the data collection process, 
use, analysis, and presentation of the statistics and performance measures. In particular, the 
study team created a manual that: 

• Identified and defined a core set of network statistics and performance measures for 
public libraries; 

• Introduced a number of additional statistics and performance measures for libraries to 
consider if appropriate to their particular management, decision making, or data needs 
situations; 

• Described the methodologies through which to collect and report the statistics and 
performance measures; 

• Suggested a number of supplemental data collection activities, including focus groups 
and interviews, for libraries to consider when trying to collect qualitative, anecdotal data 
regarding their network service and resource usage; 

• Described issues and recommendations for libraries to consider when deciding which 
statistics and performance measure to use under various decision making or other 
circumstances; 

• Suggested a number of data management strategies for libraries to consider when 
collecting usage statistics in general and network resource and service usage statistics in 
particular; and 

• Provided tutorials as to how to integrate online vendor database statistics across different 
vendors into a single database for analysis purposes. 

The American Library Association published the manual: Bertot, J.C., McClure, C.R., and Ryan, 
J. (2001). Statistics and performance measures for public library networked services. Chicago, 
IL: American Library Association. ISBN: 0-8389-0796-2. 

Dissemination Activities 

As identified throughout this report, the study team conducted a number of project 
dissemination activities that included presentations, workshops, panel discussions, and the 
production of a network statistics and performance measure manual. The study team also 
maintained a project website (initially http://www.albany.edu/~imlsstat/, now located at 
http://www.ii.fsu.edu/Albany_IMLS_Webfiles/public_html/) on which the study team published 
all project-related documents, reports, etc. The study team continues to maintain the website and 
will build upon that website as the team begins to work on the development of a national 
network statistic and performance measure data collection model. The study team is also 
presenting an all-day workshop, in conjunction with ALA Midwinter (January 14, 2001) in 
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Washington, D.C. to providing training and dissemination on how to use, administer, and report 
the statistics and performance measures outlined in the manual. 

FUTURE DIRECTIONS 

The importance of the funding IMLS provided to conduct this research project was 
significant, as it: 

• Lifted the research efforts in this area into a national forum; 

• Underscored the need for a standardized approach to public library network statistics and 
performance measures; 

• Identified the need to incorporate network statistics and performance measures into 
existing public library data collection efforts so as to reflect accurately the services public 
libraries provide; and 

• Laid the foundation for an evolving dialog in the library community as to how best to 
capture and present its involvement with network services and resources provision, uses, 
and usage. 

The efforts in which the study team engaged, though substantial, indicate that these are only the 
beginning steps of a long-term research effort in which statistics and performance measures will 
evolve over time to meet the data collection and management needs of public libraries, state 
library agencies, and policy making entities. In short, there is more work to be done. 

Building a National Data Collection System 

With continued support from IMLS, the study team will pursue a means through which to 
develop a national data collection system for public library network statistics and performance 
measures. This effort builds upon the foundation laid through this initial research project that 
established a core set of statistics and measures. 

Research by the authors - as well as numerous formal and informal interviews, focus groups, 
and discussions with library leaders, researchers, and policy makers - suggests that there may be 
numerous approaches to the development of a national network statistics and performance 
measure collection, reporting, and analysis system. These include: 

• Extending the current National Center for Education Statistics (NCES), National 
Commission on Libraries and Information Science (NCLIS), state library agency, and 
public library Federal State Cooperative System (FSCS) collaborative approach for 
annual public library data collection. In this model, public library data on selected 
statistics are passed from public libraries to state library agencies up to NCES for 
compilation, analysis, and reporting. All 50 states plus the District of Columbia and U.S. 
Territories participate in the process. 

At present, there are approximately 50 data elements collected and reported through the 
FSCS process (e.g., operating budgets, FTEs, circulation). It is necessary to propose new 
elements through an administrative procedure, and element adoption requires the vote of 
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at least 26 state data coordinators (personnel located in state library agencies) with a 
three-year phase-in on inclusion by all 50 states, Washington, D.C., and U.S. Territories. 
The adopted elements then go into the following year’s survey form for collection. The 
time from element vote to adoption to collection can be as long as three years. 

Over the last several years, the FSCS group undertook several efforts to adopt a variety of 
network statistics. All have failed to get the necessary votes for adoption. 

• Developing a lead states and libraries approach to data collection and reporting. For a 
variety of reasons, it may not be feasible for all public libraries and state library agencies 
in the nation to simultaneously adopt and report data on a set of network statistics and 
performance measures. Flowever, research by the authors demonstrates that there are a 
number of states (20+) that indicate their willingness and/or desire to collect at least a 
core set of network statistics and performance measures from the public libraries within 
their states. The same research shows that, while a state library agency or a number of 
public libraries within a state may not be willing or able to collect network statistics, lead 
public libraries within states find it imperative to collect such data for a variety of 
decision making, management, and reporting purposes. In this model, lead public 
libraries and state library agencies adopt, collect data, analyze data, and report data on a 
core set of network statistics and performance measures. The lead state library agencies 
and public libraries also serve as an incubator for developing, defining, and reporting new 
network statistics and performance measures. 

• Creating an ongoing sampling design to generate national estimates. This model 
employs a sampling approach for a variety of data collection activities to use with public 
libraries, state library agencies, and library network consortia. The intent of this 
approach is to develop a sample that would enable the generation of national estimates of 
core set of network statistics and performance measures from public library, state library 
agency, and library network consortia. Such a model would permit the targeting of 
network statistics appropriate to the level of data collection - library, state library agency, 
library consortia - as well a framework for modifying or creating new statistics and 
performance measures on an as needed basis. It would be possible to engage in the data 
collection process on a regular (e.g., annual, biannual) and/or ad hoc (e.g., as necessary) 
basis. 

• Adopting a combination approach to network statistic and performance measure data 
collection. The above data collection models are not mutually exclusive. Rather, it is 
possible to combine aspects of the FSCS, lead state/library, and sampling approaches to 
collect, analyze, and report public library network statistics and performance measures so 
as to provide nationally aggregated network statistical data. 

A key aspect of the future research project is to determine which model or aspects of the above 
models - including models and/or approaches not yet identified - are appropriate under what 
circumstances for the development, definition, collection, analysis, and reporting of national 
public library network statistics and performance measures. 

While it is not possible at this time to determine which model, or aspects of the various 
models, is most appropriate for collecting national network statistics and performance measures, 
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it is possible to identify some key characteristics that the national approach should have based on 
current research by the authors. At a minimum, these characteristics include: 

• Creating a fast response approach to the development, collection, analysis, and reporting 
of network statistics and performance measures. A key criticism of the FSCS process is 
the time lag between the development of data elements and the eventual reporting of 
those elements. For a variety of reasons, it can take four years under the current FSCS 
process from development to reporting of statistics (to be fair, the FSCS group undertook 
changes in its bylaws recently to expedite the element adoption and reporting process). 
By the time the NCES releases the public library data reports, the data are often outdated. 
This is particularly problematic in the networked environment in which any network 
statistics and performance measures will likely remain relevant for two-three years. 

• Fostering an environment of constant change. Gone are the days of statistics and 
performance measures that last for decades. The networked environment is such that 
change in technologies and the implementation of those technologies is rapid. Thus, the 
statistics and performance measures that capture network data will necessarily undergo 
constant modification. It is imperative, therefore, that the model for national library 
network statistics and measures foster an environment of flexibility, change, and 
creativity in the creation, collection, and reporting of statistical data. 

• Implementing a reasonably burden free data collection and reporting process for public 
libraries, state library agencies, and library consortia. It is clear that data reporting 
requirements imposed on public libraries are arduous. It is also clear, however, that 
network usage statistics are increasingly important to professionals, researchers, and 
policy makers. Thus, it is necessary to develop a data collection and reporting system 
that provides maximum benefit for minimal effort. 

• Working with non-library partners to gain access to library network data. Increasingly, 
key network usage data is out of the public library, state library, and library consortia 
domain. Examples include online database usage, Internet service provider (ISP), and 
telecommunications carrier (e.g., bandwidth consumption) data. It is critical to the 
measurement of library network services that the national data collection activities 
develop reporting partnerships with, minimally, the online database vendor, ISP, and 
telecommunications carrier communities. 

Undoubtedly, there are other characteristics necessary for a national public library network 
statistics and performance measure data collection system, but the above are key. 

Collaboration with Key Research Efforts 

The study team continues to work with a number of research efforts in the area of network 
statistics and performance measures, including those of NCLIS, ARL, ICOLC, and the National 
Information Standards Organization (NISO). It is essential for this collaboration to occur if there 
is to be any hope of generating, where possible, standardized statistics, definitions, and data 
collection techniques. 

As part of the national statistics model development process the study team will pursue - to 
the extent possible - collaborative efforts so as not to duplicate works in progress. 
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Evolving Statistics and Performance Measures 

Clearly, the statistics and performance measures will need to evolve over time to reflect 
changes in library operating environments as well as changes in technology. As such, the 
researchers, and others, will need to facilitate the process through which statistics and 
performance measures for the networked environment undergo periodic review for necessary 
modification and/or whole cloth development. The national model described above will need to 
incorporate a review process that addresses this issue. 

Moving towards Outcomes Measures 

As identified in the IMLS report Perspectives on Outcome Based Evaluation for Libraries 
and Museums (IMLS, 1999), outcomes relate to benefits and/or changes that individuals, groups, 
or institutions accrue after having been exposed to a particular program or effort. The statistics 
and performance measures developed through this study and described in this report would 
produce outcomes at the public library institution level. It is, however, premature to determine 
the impacts, benefits, or other outcomes that public libraries would gain until public libraries 
collect, analyze, and present the statistics and performance measures on a regular basis. Possible 
outcome measures for public libraries would be: 

• Increased staff funding for network service and resource-related library functions; 

• Additional public access workstations, thus reducing the “per capita” ratios; 

• Increased reference activity with the availability of online reference services; 

• Increased collections usage with the availability of online serials and other database 
material; and 

• Increased visitors to the library, with the inclusion of virtual visitors to the walk-in count. 

It is important to note that these outcomes (and other possible outcomes) will require a period of 
collection at that library level prior to their ability to be measured on a regular basis. 

The current network statistics and measures focus on institutional services. This is largely 
due to the nature of developing national statistics and performance measures - there are nearly 
9,000 public library systems throughout the United States, and each is different in terms of size, 
technology infrastructure, and other key variables related to network statistics and performance 
measures. Given the complexity of the public library community, the first step in outcomes 
measures focused on the institution. There is, however, a need to consider outcomes measures at 
the individual user or group users of library services. Such efforts will undoubtedly follow once 
libraries have a greater familiarity with network service and resources measurement activities. 
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CONCLUSION 

Over the course of the project a significant range of accomplishments have occurred that 
have moved the profession forward regarding statistics, performance measures, assessment 
techniques, and national approaches for collecting and reporting public library data in a 
networked environment. These accomplishments occurred not only because of the funding of 
the project from the Institute for Museum and Library Services, but also because of the extensive 
number of people in the library/information science field that wanted to be part of these 
accomplishments. 

The study team continues to be very pleased and gratified with the level of interest and 
support for this project and our efforts to design a national data collection and reporting strategy 
for public library networked services and resources in the future (IMLS funded project 2000- 
2001). Despite this gratification, there is considerable work yet to be done in this area. To some 
degree, the technical work on defining and field testing statistics and measures may be easier 
than the political and coordinating work among various stakeholder groups. 

As work on this project suggests, there is a very large range of stakeholder groups that all 
have differing points of view as to which statistics and performance measures should be 
collected and reported. These groups include local libraries, regional consortia, state libraries and 
state consortia. Also interested are national groups such as IMLS, ICOLC, NISO, the FSCS, and 
NCLIS. And of course there are a range of private sector database vendors that are quite 
interested in these efforts to develop statistics and performance measures for the networked 
environment. 

Continued success in this area of research will depend on the ability of these various 
stakeholder groups to coordinate their development work and to reach agreement not only about 
specific statistics and performance measures but how best to present and report those data. The 
study team will continue working with these various groups, it will continue to disseminate the 
results from the project reported here through a number of state, local, and national conferences, 
and it will continue to develop, refine, and test additional statistics and performance measures 
related to the networked environment. 
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APPENDIX A - SAMPLE DATA COLLECTION TOOLS 
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Network Measures Questionnaire 

Name: E-mail: 

Title: 

Phone: Fax: 

Address: 

1) Please rate usefulness to you of the existing data collected about your library’s networked services/resources? 

12 3 4 5 

Not at all Useful Very Useful 

Useful 

2) Please rate the usefulness to you of the existing data being collected on the state’s public libraries’ networked 
services/resources? 

12 3 4 5 

Not at all Useful Very Useful 

Useful 

3) Please rate the need to expand or revise existing efforts to collect and use data on public libraries’ networked 
information services/resources. 

12 3 4 5 

No Need A Top Priority 

4) Data on public library network resources and services is useful to me if (be as specific as possible): 



5) Think of three important recurring occasions over the past year when you used or could have used data about one 
or more of the state’s public libraries’ network resources and services? 



What data did you need? 


When did you need the data? 


Why did you need the data? 





















6) Are there 2-3 key public library opinion leaders in the state knowledgeable about public library network resources 
and services measures who we should interview via telephone or e-mail? 

Name E-mail Phone 



The study team will compile a set of recommended potential network measures upon completion of the site visits. 
The study team intends to: 

• Assess site visit participant’s preferences for these proposed network measures via an e-mail survey; 

• Conduct tests for problems in collecting and presenting the resulting recommended measures; and, 

• Produce and test a field manual that introduces the recommended measures, their efficient collection, 
interpretation, and effective use. 

In order to complete these tasks the study team would like your help as follows. 

7) Would you agree to participate in an e-mail survey of preferences for specific potential network measures at a 

future date? Yes No 

8) Would you agree to participate in testing the collection and presentation of a draft set of network measures at a 

future date? Yes No 

9) Would you agree to review a draft copy of a national network statistics users’ manual for us at a future date? 

Yes No (Continued on the next page) 
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10) Please provide us with your recommendations and comments on the following performance measures. 



Measure 


Definition 


Recommend 

Yes/No? 


Comments 


Public access 
library 

workstations per 
capita 


The number of public access Internet 
workstations in the library divided by the 
population of the library’s legal service 
area. Further refinements: Workstations 
connected at 56k or less v. workstations 
connected at greater than 56k; 
Graphical/text terminals; Filtered/non- 
filtered workstations. 






Electronic 
reference 
transactions per 
capita 


The number of reference transactions 
conducted via email or via a library’s 
Website divided by the population of the 
library’s legal service area. 






Response time to 
electronic 
reference 
transactions 


The average time (as measured in some 
unit such as days or hours) that users 
receive answers to electronic reference 
questions. 






Web visits per 
month 


A visit is a user who visited a Web site, 
regardless of the number of pages, or 
elements he or she viewed. If a user 
looked at sixteen pages and fifty-four 
graphics while at a Web site, that user 
registered one visit on the Web server. A 
visit is usually determined by a user’s IP 
address, which can be misleading due to 
Internet Service Providers (ISP’s and 
Firewall’s or Proxy Servers). Thus, this 
measure is an estimate of the visits. 






% of annual 
budget for 
information 
technology 
expenditures 


All expenditures for information 
technology (example list of expenditures 
to be developed) divided by the total 
annual expenditures of the library. 






% of annual 
materials budget 
for electronic 
resources 


All expenditures for electronic resources 
(example list to be developed) divided 
by the total annual materials 
expenditures. 






Public Internet 
training per 
month 


The total hours per month that library 
staff have provided Internet training in a 
formal or pre-scheduled session. 






Overall public 
access 

workstation use 


The % per month (or some measure of 
time e.g., week or day) that public access 
Internet workstations are in use divided 
by number of hours of library operation 
during that time period. 






Saturation of 

network 

use/resources 


The percentage of network resource use 
(e.g., number of modems in modem in 
use; telnet/remote logins; bandwidth 
consumption) per measure of time (as 
measured in some increment of time 
such as hour, day, week, or month). 







11) Please make additional comments here: 
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Developing Public Library Statistics for the Networked Environment 
FSCS Coordinator, Knowledgeable Staff, & Project Liaison Data Collection Form 

What Network Measures Do You Need 

Focus for a moment on your need for network measures in order to do your job better what are the recurring events, 
recurring demands from significant groups, recurring purposes, etc. 

State Library Itself 

1) Name three important recurring occasions over the past year when you used or could have 
used data about the State Library’s network resources and services? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















The State’s Public Libraries 



2) Name three important recurring occasions over the past year when you used or could have 
used data about one or more of the state’s public libraries’ network resources and services? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















The Nation’s Public Libraries 



3) Name three important recurring occasions over the past year when you used or could have 
used data about one or more of the nation’s public libraries’ network resources and services 
(other than your own library)? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















4) Why is national data, e.g., data about other public libraries and other state libraries useful to you? How do you, 
would you, use this data? 



5) What network measures does the State Library collect at present? Interviewer check to see if we have all the data 
requested in the data call. Probe: Which measures do you find particularly useful, if any? Why? 




43 



Bertot, McClure, & Ryan 



53 



November 16, 2000 



Developing Statistics and Performance Measures for the Networked Environment: Final Report 



6) What network resources and services lend themselves to measurement and data collection activities? 
Resource or Service Measure 



7) Summarizing, data on public library network resources and services is useful to you if (be as specific as possible): 
Probe: Helps politically: to justify present activities, seek new funds 

Improves management: To plan better, public library scanning function 

By type of audience: legislature, governor, public library directors, general public 

By recurring event: budget 

Other: 
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Assessment of Potential Network Measures 



8) Refer to the briefing paper distributed prior to the study team’s visit. Interviewer use the Assessment of Potential 
Network Measures chart. For each potential measure identified: Is the measure feasible, who would use, is it useful, 
problems and issues you foresee? 



9) Are there network measures that should be dropped or added to this proposed list? Interviewer use blank chart 
pages? Refer back to the initial needs identified by person being interviewed. 



Techniques for Statewide Data Collection, Analysis, and Reporting 

10) Who will the State Library use to collect, interpret, and present network measures? Does this represent a 
commitment of new staff, a re-deployment of existing staff (something is no longer done), or no change in staffing? 



11) Can the feasible and useful network measures identified be produced by vendors or database providers? 
. Proposed Network Measure Who Does/Could Collect? How? 



12) (a) What steps do you take currently to insure the reliability and validity of data collection activities? (b) What 
future steps are feasible? 



13) What are the current steps you take to collect network measures? Interviewer, it might be useful to suggest to the 
person interviewed start with a particular month and go through the year. 

Step Problem or Issue 



14) How can the burden of data collection of network measures be reduced? 

15) How should the network measures that you and others identified as both feasible to collect and usable be 
presented to you and others for maximum impact and use? 

Measure How Used? How presented? 

16) Can groupings of network measures be developed for target audiences, for example, network measures for rural, 
suburban, urban public libraries? Good idea, bad? Suggested ways of grouping? 



17) Can you readily connect to the Internet via a workstation at your desktop? Yes No 

1 8) Do you use spread sheet or statistical packages? Excel SPSS SAS 

Other spreadsheet (Which one:) Other statistical package 

19) Does the State Library collect statistics from you using the Internet? Yes No 

Coordination Across Levels 

20) Can you suggest mechanisms to better coordinate federal, state, and local collection of 
network measures? 
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Developing Public Library Statistics for the Networked Environment 
Public Library Data Collection Form 

What Network Measures Do You Need 

Focus for a moment on your need for network measures in order to do your job better what are the recurring events, 
recurring demands from significant groups, recurring purposes, etc. 

Your Public Library (System) Itself 

1) Name three important recurring occasions over the past year when you used or could have 
used data about your library’s network resources and services? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















The State’s Public Libraries 



2) Name three important recurring occasions over the past year when you used or could have 
used data about one or more of the state’s public libraries’ network resources and services? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















The Nation’s Public Libraries 



3) Name three important recurring occasions over the past year when you used or could have 
used data about one or more of the nation’s public libraries’ network resources and services 
(other than your own library)? 



When did you need the data? 


What data did you need? 


Why did you need the data? 





















4) Why is national data, e.g., data about public libraries in other states and national figures useful to you? How do 
you, would you, use this data? 



5) What network measures does your library collect at present? Interviewer ask for copies. Probe: Which measures 
do you find particularly useful, if any? Why? 
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6) What network resources and services lend themselves to measurement and data collection activities? 
Resource or Service Measure 



7) Summarizing, data on public library network resources and services is useful to you if (be as specific as possible): 
Wait then probe: Helps politically: to justify present activities, seek new funds 

Improves management: To plan better, public library scanning function 

By type of audience: legislature, governor, public library directors, general public 

By recurring event: budget 

Other: 
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Assessment of Potential Network Measures 



8) Refer to the briefing paper distributed prior to the study team’s visit. Interviewer use the Assessment of Potential 
Network Measures chart. For each potential measure identified: Is the measure feasible, who would use, is it useful, 
problems and issues you foresee? 



9) Are there network measures that should be dropped or added to this proposed list? Interviewer use blank chart 
pages? Refer back to the initial needs identified by person being interviewed. 



Techniques for Data Collection, Analysis, and Reporting 

10) Who will your library use to collect, interpret, and present network measures? Does this represent a commitment 
of new staff, a re-deployment of existing staff (something is no longer done), or no change in staffing? 



11) Can the feasible and useful network measures identified be produced by vendors or database providers? 
Proposed Network Measure Who Does/Could Collect? How? 



12) (a) What steps do you take currently to insure the reliability and validity of data collection activities? (b) What 
future steps are feasible? 



13) What are the current steps you take to collect network measures? Interviewer, it might be useful to suggest to the 
person interviewed start with a particular month and go through the year. 

Step Problem or Issue 



14) How can the burden of data collection of network measures be reduced? 



15) How should the network measures that you and others identified as both feasible to collect and usable be 
presented to you and others for maximum impact and use? 

Measure How Used? How presented? 



16) Can groupings of network measures be developed for target audiences, for example, network measures for rural, 
suburban, urban public libraries? Good idea, bad? Suggested ways of grouping? 

17) Can you readily connect to the Internet via a workstation at your desktop? Yes No 

18) Do you use spread sheet or statistical packages? Excel SPSS SAS 

Other spreadsheet (Which one:) Other statistical package 

19) Does the State Library collect statistics from you using the Internet? Yes No 

Coordination Across Levels 

20) Can you suggest mechanisms to better coordinate federal, state, and local collection of network 
measures? 
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Problems & Issues 






Usability? 






Who Needs? 






Feasibility? 






Definition 


• Report the type of 
Internet access 
available (none, 
dial-up, 56K, Tl, 
ISDN, other) 

• Report the type of 
Internet services 
available (none; e- 
mail; newsgroup; 
graphical web- 
browsing; text- 
based web- 
browsing; FTP; 
telnet; login; other) 


• Count the total 
number of printers 
available to the 
public 

• Report the ratio of 
the number of 
computers which 
have access to 
printers to the 
number of printers 
available to the 
public 


1 Data Element 


Internet Access 
Available to the 
Public 


Printers Available to 
the Public 
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Problems & Issues 






Usability? 






Who Needs? 






Feasibility? 






Definition 


Definition: The total 
hours per month that 
library staff have 
provided Internet 
training in a formal or 
pre-scheduled session. 




1 Data Element 


Public Internet 
training per month. 
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Problems & Issues 






Usability? 






Who Needs? 






Feasibility? 






Definition I 






1 Data Element 1 
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Preliminary Network Statistics 





Collections 


CD-ROMs 


Definition: CD-ROM refers to compact discs that do not solely contain audio 
information and are available for public use. 

• Count the number of physical units 

• Count the number of unique titles 

• Count the number of titles contained within each subscription 

• Count the number of customers who can simultaneously use each title at once 


Electronic subscription services 


Definition: This element includes both full-text and index (e.g., EBSCO, Gale, Wilson) 
electronic subscriptions that are available for public use. 

• Count the number of discrete, vendor-based systems subscriptions 

• Count the number of titles contained within each subscription 

• Count the number of customers who can simultaneously use each vendor 
subscription 


Software packages 


Definition: Software packages means software such as Microsoft Word, Microsoft 
Excel, or any other software that allows for the performance of similar tasks and that are 
available for public use. 

• Count the number of physical units 

• Count the number of unique titles 

• Count the number of customers who can simultaneously use each title at once 

Equipment and Access 


Computers available to the public 


• Count the number of computers, in total, available 

• Count the number of dumb terminals available 

• Count the number of Mac and PC computers available 

• Count the number of computers which provide access to the Internet 

• Count the number of computers which are connected to CD-ROM resources 

• Count the number of computers which provide access to software packages 

• Count the number of computers which provide access to electronic subscription 

services 

• Count the number of computers which provide access to the OPAC 


Internet access available to the 
public 


• Report the type of Internet access available (none, dial-up, 56K, T1 , ISDN, other) 

• Report the type of Internet services available (none; e-mail; newsgroup; graphical 
web-browsing; text-based web-browsing; FTP; telnet; login; other) 


Library home page services 


• Report the type of information and services provided by the library's web 
site: 

> no web site; 

> description of library services; / 

> access to library’s catalog or to consortium/regional catalogs; 

> access to subscription databases restricted to library patrons only; 

> access to subscription databases not restricted to library patrons only; 

> a selection of links; 

> capability for public to e-mail reference questions to staff) 


Printers available to the public 


• Count the total number of printers available to the public 

• Report the ratio of the number of computers which have access to printers to the 
number of printers available to the public 
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Preliminary Network Statistics (cont'd) 





Costs 


Computer-Related Hardware 
Available to the Public 
Expenditures 


• Report expenditures made for computer-related hardware that is made 
available to the public. This can include: 

> Expenditures for computers, printers, modems, or other associated 
hardware designed to provide access to electronic services to the 
public 


Electronic Access Expenditures 


• Report all operating expenditures from the library budget associated with 
access to electronic materials and services. Include expenditures for 
equipment used to run information service products when it cannot be 
separated from the price of the product 

• Report expenditures for services provided by national, regional, and local 
bibliographic utilities, networks, consortia, and commercial services 

• Report all fees and usage costs associated with such services as electronic 
subscription services or electronic document delivery. 


Electronic Format Expenditures 


• Report operating expenditures for materials in electronic format that are considered 
part of the collection, whether purchased or leased, such as 

> CD-ROMs (using the definition in the Collections section); 

> magnetic tapes; and 

> Magnetic disks that are designed to be processed by a computer or similar 
machine 

• Report expenditures for computer software for patron use here 


Maintenance of Computer- 
Related Hardware Available to 
the Public Expenditures 


• Report expenditures for maintenance of computers, printers, or other computer- 
related hardware designed to provide access to electronic services to the public 


Telecommunications 

Expenditures 


• Report all expenditures relating to telecommunications, excluding voice 
communications, and including Internet service provider expenditures 

Use 


Accesses to the library's Web 
pages 


• Count the number of accesses to the library's top 5 web pages during a 

representative one-week period. The top 5 web pages is determined by analyzing 
the logs of all the pages and ranking the pages in order of accesses 


Electronic subscription services 
use 

by the public 


• Count the number of searches performed on computers available to the public 
within the library during a representative one-week period 

• Count the number of searches performed on computers remote to the library during 
a representative one-week period 


OPAC use by the public 


• Count the number of OPAC searches performed on computers available to the 
public within the library during a representative one-week period 

• Count the number of OPAC searches performed on computers remote to the library 
during a representative one-week period 
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Preliminary Performance Measures 




Public access library workstations per capita 


Definition: The number of public access Internet 
workstations in the library divided by the population of 
the library’s legal service area. Further refinements: 

• Workstations connected at 56k or less versus 
workstations connected at greater than 56k; 

• Graphical/text terminals; 

• Filtered/non- filtered workstations. 


Electronic reference transactions per capita 


Definition: The number of reference transactions 
conducted via email or via a library’s Website divided 
by the population of the library’s legal service area. 


Response time to electronic reference transactions 


Definition: The average time (as measured in some 
unit such as days or hours) that users receive answers 
to electronic reference questions. 


Web visits per month 


Definition: A visit is a user who visited a Web site, 
regardless of the number of pages, or elements he or 
she viewed. If a user looked at sixteen pages and fifty- 
four graphics while at a Web site, that user registered 
one visit on the Web server. A visit is usually 
determined by a user’s IP address, which can be 
misleading due to Internet Service Providers (ISP’s and 
Firewall’s or Proxy Servers). Thus, this measure is an 
estimate of the visits to the Website. 


Percentage of annual budget for information 
technology expenditures 


Definition: All expenditures for information 
technology (example list of such expenditures to be 
developed) divided by the total annual expenditures of 
the library. 


Percentage of annual materials budget for electronic 
resources 


Definition: All expenditures for electronic resources 
(example list of such expenditures to be developed) 
divided by the total annual materials expenditures. 


Public Internet training per month 


Definition: The total hours per month that library staff 
have provided Internet training in a formal or pre- 
scheduled session. 


Overall public access workstation use 


Definition. The percentage per month (or some 
measure of time such as week or day) that public 
access Internet workstations are in use divided by the 
number of hours of library operation during that time 
period. 


Saturation of network use/resources 


Definition: The percentage of network resource use 
(e.g., number of modems in modem in use; 
telnet/remote logins; bandwidth consumption) per 
measure of time (as measured in some increment of 
time such as hour, day, week, or month). 



Bertot, McClure, & Ryan 74 


November 16, 2000 

103 



Developing Statistics and Performance Measures for the Networked Environment: Final Report 



APPENDIX D - GENERAL FIELD TEST PRELIMINARY NETWORK STATISTICS 

AND PERFORMANCE MEASURES 
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INTRODUCTION 

The study team has engaged in numerous data collection activities to develop national public 
library statistics and performance measures for the networked environment between January 1999 and 
September 1999. These activities include (see http://www.albany.edu/~imlsstat/ for detailed descriptions 
and findings from the study’s research activities): 

• Establishing a project advisory committee comprised of library professionals, 
administrators, executives, data collection and statistics coordinators, and vendors; 

• Reviewing extensively electronic network statistics and performance measure literature; 

• Reviewing extensively available network statistic and performance measure data 
collection efforts at the state library agency level (for all states that publish their data on 
the Web, not just the six state study participants); 

• Conducting in-depth surveys with the six participating state library agencies regarding 
state library electronic network statistics and performance measure data collection 
activities and online database vendor-supplied statistics; 

• Developing preliminary network statistics and performance measures based on the data 
collection efforts; 

• Pre-testing the preliminary network statistics and performance measures with the six state 
study participants. The pre-test also served to inform the study team about the research 
approach and data collection efforts for the remaining five state study participants; 

• Conducting focus groups and individual interviews with public and state library leaders 
regarding the proposed statistics and performance measures; 

• Presenting current findings from the project at national, state, and local conferences and 
meetings; 

• Revising the statistics and performance measures based on the site visits and study data 
collection activities; and 

• Pre-testing the revised statistics and performance measures with the Federal-State 
Cooperative System Working Group on Statistics meeting in Washington, D.C. in 
September 1999. 

Based on the data collection activities above, the study team developed a final list of statistics 
and performance measures to field test (see Tables 1 and 2). 

This document outlines the network statistics field test methodology and timeline. Upon 
completion of the field test, the study team will begin the to develop and test a network statistic 
and performance measure manual for use by librarians, researchers, and policy makers. 
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METHODOLOGY 

The site visits to the six participating states of Delaware, Maryland, Michigan, North 
Carolina, Pennsylvania, and Utah demonstrated that no one state collects, maintains, or 
disseminates its public library statistics in the same manner. Moreover, each state, and the public 
libraries within those states, has a substantially different information technology infrastructure 
and implementation of networked services and resources. For example: 

• Maryland has a statewide network (Sailor) through which it provides access to networked 
information resources (http://www.sailor.lib.md.us/); 

• Delaware has a state-library provided Web site (DelAWARE) through which it provides 
access to networked information resources (http://www.lib.de.us/). 

• Pennsylvania does not have a statewide network. However, the state recently entered 
into a statewide licensing agreement with online database vendors to provide its Power 
PA service; 

• North Carolina has a statewide consortial arrangement, NC Live, in which state library, 
public libraries, and public and private universities participate. NC Live provides access 
to numerous online databases (http://www.nclive.org/); 

• Michigan engages in online Web-based public library data collection activities; and 

• Utah recently contracted with Bibliostat, a software product produced by Management 
Dynamics that specializes in the collection and analysis of public library data, to collect 
and maintain Utah public library statistics (http://www.bibliostat.com/). 

As such, the study team determined that no one research approach would be appropriate for the 
for the field test. Rather, the study team determined that a methodology that embraced the 
differences and unique aspects of the state public library networked service and resource 
configurations would better inform the field-testing of the statistics. 

Goals of the Field Test 

The goals of the field test are to: 

• Finalize the definitions of the statistics and performance measures; 

• Test various data collection and reporting methods; 

• Work with online database vendors to develop a standardized set of statistics, reporting 
techniques, and data provision across vendors; 

• Determine a final core set of network statistics and performance measures; and 

• Identify issues and make recommendations for the collection of network statistics. 

The primary focus of the field test will be on the statistics, as the statistics form the basis for the 
performance measures. The study team considers it critical, therefore, to concentrate on validating the 
network statistics. 
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Field-Testing In-library Statistics 

While somewhat of an oversimplification, it is possible to divide the network statistics into two 
broad categories of in-library statistics and external statistics. In-library statistics are data that are largely 
under the direct control of the library and include (see Table 3) 

• Public access workstations. Count of the number of public Internet access workstations, 
workstation usage, and maximum speed of Internet connection; 

• Databases. Total expenditures for online electronic material; 

• Electronic services. Count of the number of electronic reference transactions; 

• Virtual visits. Website access or remote login count; and 

• Training. Count of the number of users trained and the number of staff trained per year. 

Most of the field test participants will focus on data collection for the above statistics. 



Table 3. Proposed In-Library Statistics for Field-Testing. 


Public Access Workstations 


Data Element/ 
Statistic 


Definition 


Core/ 

Optional 


Proposed Method 
of Collection 


Proposed State(s) 
to Collect 
Information 


# Public access 
workstations 


Count of the number of public access 
graphical workstations that connect to 
the Internet. 


Core 


Total count 


• Delaware 

• Maryland 

• Pennsylvania 

• Utah 


Public access 
workstation usage 


Measure of public access workstation 
usage through count of users. 


Optional 


Sampling at a period 
to be determined 
such as 2 weeks 
every quarter. 


• Delaware 

• Maryland 

• Pennsylvania 

• Utah 


Maximum speed of 
public access 
Internet 
workstations 


Indication of the maximum speed of 
public Internet access, e.g., 56kbps, 
ISDN, Tl. 


Core 


Identify fastest public 
access connection 
speed. 


• Delaware 

• Maryland 

• Pennsylvania 

• Utah 


Databases 


Electronic material 
expenditures 


Total expenditures for online 
electronic material subscription 
expenditures (Internet-based 
subscription services). These may 
include one-time start-up costs as well 
as recurring costs. 


Core 


Total expenditures 
for online electronic 
materials 




Electronic Services 


# Electronic 

reference 

transactions 


Count of the number of reference 
questions/ requests received 
electronically (e.g., via e-mail). 

Note: While not all libraries provide 
this service, site visits show the need 
for such a measure to capture the 
increase over time to show a 
proportional increase. 


Core 


Separate count of e- 
mail conducted 
reference transactions 


• Delaware 

• Maryland 

• Pennsylvania 

• Utah 
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Table 3. Proposed In-Library Statistics for Field-Testing. (Cont’d). 


Virtual Visits 


Data Element/ 
Statistic 


Definition 


Core/ 

Optional 


Proposed Method 
of Collection 


Proposed State(s) 
to Collect 
Information 


# Virtual visits 


A visit is a user who visits a Web site 
regardless of the number of pages or 
elements viewed. If a user looked at 
16 pages and 54 graphic images while 
at a Web site, that user registers one 
visit on the Web server. Due to 
various Web server issues, this 
measure is an estimate of the visits to 
the Website. 

[Remote logins as well: count of 
number of remote logins (sessions) to 
non-Web-based library resources such 
as OP ACS]. 


Core 


Will need a 
standardized log 
tracker/ software to 
capture such data 
consistently. 

Also, may require 
changes in 
configuration at the 
local level to ensure 
the same data is 
collected and 
analyzed. 

This requires testing. 


• Maryland 

• Pennsylvania 

• Utah 


Training 


# Users trained 


Count of the number of users trained 
in structured and informal technology 
training sessions conducted by the 
library. Structured means a course 
with a designed curriculum intended to 
demonstrate the use of a technology 
such as the Web, Internet searching, 
personal computing, etc. Informal 
includes contacts with users by library 
staff intended to demonstrate the use 
of library workstations, aspects of the 
applications available on those 
workstations, etc. 


Core 


Debate over # users 
or contact hours 


• Delaware 

• Maryland 

• Pennsylvania 

• Utah 



The study team will use multiple methodologies, data collection techniques, and approaches 
during field test. The differences in data collection activities will take advantage of the unique 
state public library data collection activities, IT configurations, and types of network services 
provision and resources available within each of the six field test states. 

Delaware — The State Library Agency Data Collection Model 

Delaware provides access to a variety of online services through the state library-run 
DelAWARE Web site (http://www.lib.de.us/). Residents of Delaware can gain access to the 
Internet and Internet-based resources offered through DelAWARE from public access 
workstations located in public libraries as well remotely via the Web. A distinguishing factor to 
the DelAWARE service is that users must enter their library bar codes in order to: 1) Access the 
Internet from a public library public access workstation, and 2) Access the online vendor 
databases. 

For the field test, the study team envisions a collaborative data collection process between 
the state library agency and one public library. Because the state library agency provides 
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numerous network support services to the public libraries in the state, the state library agency is 
in an excellent position to act as a network statistic data collection and dissemination entity. In 
this model, the state library agency would: 

• Act as the data collection agent for public library network statistics; 

• Collect and compile network statistic data from the public libraries in the state. These 
data include the public access workstation data, electronic services data, and training 
data; and 

• Provide virtual visit and database data, as the managing agency for DelAWARE. 

Since the study team will assess the Database statistics with North Carolina and Maryland, the 
state library agency would not be asked to provide that data for the field test. 

The participating public library would provide the state library agency with: 

• Public access workstation data; 

• Electronic services (electronic reference transactions) data; and 

• Training data. 

In this field test model, therefore, the state library agency will act both as the provider of data for 
various key statistics as well as the collection and dissemination point for public library network 
statistic data in general. 

Additional possibilities, due to the barcode entry system, include breaking down barcode 
information by library system and remote v. internal access to databases and networked services. 
This would provide substantial use and user-based information. 

Maryland - Public Library and Statewide Network Model 

Maryland provides the study team with the opportunity to collect all the network statistics 
developed by the project (see the below External Library Statistics section for a further 
discussion of the Maryland field test). For the field test, the study team intends to work with: 

• Two public libraries that will collect public access workstation, electronic services, 
virtual visit (if applicable), and training data (likely to be Baltimore County and Anne 
Arundel County public libraries); and 

• Sailor Operations Center to collect virtual visit data for Sailor. 

This will provide a testing of all the proposed in-library network statistics. 

The study team will establish a field test liaison within each public library and the Sailor 
Operations Center (SOC). These liaisons will work with the study team and the state library 
project coordinator to: 

• Act as a single point of contact within SOC and the libraries; 

• Coordinate the in-library field test; 
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• Provide the study team and project coordinator with updates as to the progress and issues 
encountered during the field test; 

• Work with library staff to develop and coordinate a data collection and entry process for 
the collected statistics; and 

• Facilitate any necessary post field test follow-up. 

This approach provides a process through which the state library, participating libraries and 
SOC, and the study team are in continual contact and can address quickly field test issues. 

Michigan - Online Web-based Data Collection Model 

The Michigan state library offers its public libraries the ability to file electronically annual 
public library data via the Web (http://envoy.libofmich.lib.mi.us/). State library staff developed 
the electronic filing service to facilitate the data entry and analysis processes for public library 
statistics collected annually. 

For the field test, the study team seeks to: 

• Work with at least one public library in the state to collect public access workstation, 
electronic services, virtual visit (if applicable), and training data; and 

• Work with the state library to modify the existing Web-based data entry form for the 
network statistics so that the participating public library can enter the data electronically 
via the Web. 

In this approach, the state library provides an online data entry mechanism for the public library 
community. 

The study team will establish a field test liaison within the public library and state library. 
The public library liaison will work with the study team and the state library project coordinator 
to: 

• Act as a single point of contact within the library; 

• Coordinate the in-library field test; 

• Provide the study team and project coordinator with updates as to the progress and issues 
encountered during the field test; 

• Enter the library network statistics data into the Web-based form; and 

• Facilitate any necessary post field test follow-up. 

The state library liaison will: 

• Assist in modifying the existing online data entry forms to accommodate the electronic 
network statistics; 

• Inform the project coordinator and study team of issues encountered during the field test 
that affect the data entry process; 

• Provide the study team with the final data set from the library field test in an ASCII 
delimited file for analysis purposes; and 
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• Assist the study team during any necessary post field test follow-up. 

This process will facilitate the in-library field test as well as the Web-based data entry system. 
Pennsylvania - Distributed Model 

For Pennsylvania, the field test will involve a rural and an urban public library. In both 
cases, the libraries will collect all the network statistics — public access workstation, electronic 
services, virtual visit (if applicable), and training data — save the online database statistics. 

It is necessary to establish a field test liaison within each public library who will work with 
the study team and state library project coordinator. The public library liaison will: 

• Act as a single point of contact within the library; 

• Coordinate the in-library field test; 

• Provide the study team and project coordinator with updates as to the progress and issues 
encountered during the field test; 

• Work with library staff, the study team, and the project coordinator to develop and 
coordinate a data collection and entry process for the collected statistics; and . 

• Facilitate any necessary post field test follow-up. 

This approach will create an important coordinated data collection process given the distributed 
nature of public library data collection in the state. 

It is important to note that the state library has and continues to develop a statewide licensing 
agreement for online database services for Pennsylvania public libraries called Power PA. This 
service is still in its development stages. The study team will monitor the progress of the project 
throughout the field test period and may incorporate various aspects of the service into the field 
test. 

Utah - Commercial Data Entry Model 

Utah recently signed an agreement with Management Dynamics, Inc., the producer of the 
Bibliostat software (http://www.bibliostat.com/), to collect and analyze Utah’s public library 
statistics. It is worth noting that many other state libraries also have contracts with Management 
Dynamics for public library data analysis and other services. Also, the study team has met with 
Management Dynamics staff at various times throughout the project to review the software and 
other products under development. 

For the Utah field test, the study team intends to: 

• Work with at least one public library in the state to collect public access workstation, 
electronic services, virtual visit (if applicable), and training data (likely Salt Lake City 
Public Library). There is a need to establish a public library liaison who will 

o Act as a single point of contact within the library, 
o Coordinate the in-library field test, 
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o Provide the study team and project coordinator with updates as to the progress 
and issues encountered during the field test, and 
o Facilitate any necessary post field test follow-up; 

• Work with the state library and Management Dynamics to adapt the Bibliostat software 
for the network statistics so that the participating public library can enter the data 
electronically through Bibliostat; and 

• Work with the state library and Management Dynamics to promote the development of a 
networked/Web-based version of the data entry software so that future data collection can 
occur via the Web. 

It may also be necessary to establish a state library liaison in addition to the project coordinator 
to facilitate the necessary Bibliostat data entry modifications. 

This approach would field test a commercial/third party solution to the collection, analysis, 
and dissemination of public library statistics. 

Field-Testing External Statistics 

Libraries are not able to collect an important type of data on the usage of their electronic services 
and resources on their own. In particular, libraries do not have control over the usage data of their online 
database subscription services. Rather, the database vendors collect, maintain, and disseminate that data. 
Examples statistics include (see Table 4): 

• Databases. Count of the number of unique online database titles, a count of the number of 
sessions, queries/searches, and views. 

It will be necessary for the study team and field test participants to work with the online database vendor 
community to gain access to the above types of data. This portion of the project builds upon study team 
data collection activities performed previously to assess what information the vendor community could 
provide about electronic network resources usage (see http://www.albany.edu/~imlsstat/vendor.pdf). All 
six of the project websites subscribe to a networked licensed database, however, North Carolina and 
Maryland together provide the study team with an opportunity to review all four of the vendors (Ebsco, 
IAC/Gale, OCLC, UMI) selected in the initial review. 

North Carolina's networked database(s) are available through a statewide consortial 
agreement between the state library, public libraries, and public and private universities called 
NC Live (see http://www.nclive.org/). Maryland's networked database(s) are available through 
their statewide network, SAILOR (see http://www.sailor.lib.md.us/). The study team considers 
NC Live and SAILOR to provide an efficient forum through which to test the proposed database 
statistics with the four licensed database vendors. 



Table 4: Proposed External Library Statistics for Field-Testing. 



Databases 



Data Element/ Statistic 


Definition 


Core/ 

Optional 


Proposed 
Method of 


Proposed State(s) 
to Collect 






Collection 


Information 
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# Unique electronic titles 


Count of the number of unique online 
database titles available at the library. 
Refinements include full-text versus 
abstracted titles. 


Optional 


Total count 


• North Carolina 

• Maryland 


# Electronic network 
sessions 

(adapted from ICOLC, 

http://wwwdibrary.yale.edu/ 

consortia/webstats.html) 


Overall count of the number of sessions 
(logins) initiated to the online databases. 
Refinements include breakdowns by 
title, time of day, IP address). 


Unsure - 
field test 


Total count 
of sessions 


• North Carolina 

• Maryland 


# Electronic network 
queries/searches 
(adapted from ICOLC, 
http://wwwdibrary.yale.edu/ 
consortia/webstats.html) 


Overall count of the number of searches 
conducted in the library’s online 
databases. Subsequent activities by 
users (e.g., browsing, printing) are not 
considered part of the search process. 
Refinements include breakdowns by 
title, time of day, IP address). 


Unsure - 
field test 


Total count 
of searches 


• North Carolina 

• Maryland 


# Electronic network views 
(adapted from ICOLC, 
http://www.library.yale.edu/ 
consortia/webstats.html) 


Overall count of the number of online 
database content views (e.g., abstracts, 
full-text articles). Refinements include 
breaking down the views by user action 
- printing, e-mailing, saving to disk — 
and type of document (e.g,, PDF, text, 
image, video) 


Unsure - 
field test 


Total count 
of views 


• North Carolina 

• Maryland 



The study team will work with the project coordinators’ for Maryland and North Carolina to: 

• Request their participation during this phase of the project; 

• Explain what is expected of them during this phase; 

• Establish communication between the vendor contact person and the study team; and 

• Provide regular feedback to the study team regarding the progression of the field-testing. 

The project coordinators in both states agreed to test the proposed database statistics with their 
particular licensed database vendor(s). 

North Carolina - Online Database Consortial Model 

North Carolina, through a consortial agreement between the state library, public libraries, and 
public and private universities, developed NC Live (http://www.nclive.org/). NC Live provides 
access to a number of vendor-provided databases to North Carolina public library users as well 
as the research and higher education communities. Due to the robust nature of NC Live’s online 
vendors agreements, the study team proposes to focus on the database statistics in North Carolina 
(see the previously discussed Vendor Methodology section). In particular, the study team intends 
to: 

• Contact the vendors through NC Live and state library management structures and this 
study to explain the nature of the project, the desired data elements, data collection 
activities, and other relevant information concerning the interaction between the study 
team, NC Live, the state library, and the vendors; 

• Work collaboratively with the NC Live staff, state library staff, and vendor staff to get 
database usage data reported to the study team for analysis purposes; 
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• Test the types of data reported, the way in which the data are reported (e.g., via the Web, 
electronically via e-mail), and the format of the provided data (e.g., standard format 
across vendors for comparative analysis purposes); 

• Work with the vendors to provide data breakdowns such as by IP address; 

• Determine the types of breakdowns desired/most useful to the database licensee; and 

• Determine the issues encountered during the field test phase — e.g., difficulty of 
providing statistics from the vendors' perspective, or quality/utility of the data from the 
perspective of NC Live. 

Doing this would permit a strong focus on vendor statistics, thus identifying key issues 
concerning the availability, collection, analysis, and dissemination of online database vendor 
statistics. 

Maryland - Public Library and Statewide Network Model 

In addition to field-testing the in-library statistics with two Maryland public libraries and 
Sailor, the study team also intends to work with the state library to collect Gale/IAC Infotrac 
database usage data - a service provided via the Sailor network. The study team will: 

• Contact Gale/IAC through state library and Sailor Operations Center (SOC) management 
structures and this study to explain the nature of the project, the desired data elements, 
data collection activities, and other relevant information; 

• Work collaboratively with SOC staff, state library staff, and vendor staff to get database 
usage data reported to the study team for analysis purposes; 

• Test the types of data reported, the way in which the data are reported (e.g., via the Web, 
electronically via e-mail), and the format of the provided data (e.g., standard format 
across vendors for comparative analysis purposes); 

• Work with the vendors to provide data breakdowns such as by IP address; 

• Determine the types of breakdowns desired/most useful to the database licensee; and 

• Determine the issues encountered during the field test phase — e.g., difficulty of 
providing statistics from the vendors' perspective, or quality/utility of the data from the 
perspective of Gale/IAC. 

This approach will provide the study team with useful information regarding the vendor issues in 
providing a variety of database usage data in a number for formats. 

Collecting the Proposed Statistics 

The study team is asking each licensed database vendor to attempt to collect the three 
statistics: number of electronic sessions, number of electronic queries/searches; and number of 
network views. Based upon the first vendor analysis conducted by the study team, some of the 
licensed database vendors are already collecting the proposed statistics. However, the licensed 
database vendors’ statistic definitions do not always coincide with those developed by the study 
team - and others such as the International Coalition of Library Consortia (ICOLC) — through 
the various project data collection activities. As such, the study team will ask licensed database 
vendors to: 
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• Collect the statistics as defined by the study team (and the ICOLC); 

• Provide the methodology behind collecting the statistics; 

• Include the statistical refinements (see Table 3) where applicable; and 

• Tell the study team where the statistics are available (vendor website, faxed, emailed to 
subscriber). 

Each vendor will be asked to provide the data as defined by the statistics for a two week period 
(or another time period that is easiest for the vendors) to coincide with the two week in-library 
data collection phase of the field test. During this time, the study team expects to be in contact 
with the project coordinators in Maryland and North Carolina, the management teams for the 
networked resources in each state, and the licensed database vendors. By having an open forum 
for communication, the intent is to identify and resolve key data collection, definitional, and 
other issues. 



Issues/Difficulties Encountered During Field-Testing^ 

The study team is aware that there are a number of issues that will arise during the field- 
testing. One critical concern for the study team is the willingness of the vendors to participate in 
the field study process. The study team has identified a number of factors that affect a vendor’s 
willingness to participate in the field test, including: 

• Licensed database vendor(s) statistic(s) already match what the study team proposed; 

• Study team's proposed statistic definitions are difficult to collect; 

• Licensed database vendor(s) want their statistic(s) to remain unique; and 

• Licensed database vendor does not want to participate in the field-testing. 

The study team will work with the vendors, project liaisons, and online database management 
teams in the field test sites to gain vendor support for the project to the extent possible. 

FIELD TEST TIMELINE 

The study team anticipates conducting the field test for two weeks during the October 15 
through November 15, 1999 timeframe. It is likely that the actual data collection activities 
would occur between November 1 and November 15, 1999. This would provide the field test 
participants with one month to finalize the arrangements necessary for the field tests. 

At the end of the two-week period, the study team will conduct follow-up interviews and 
discussions as necessary to clarify issues that arose during the field test. 



MANAGEMENT OF THE FIELD TEST 

The study team will work directly with field test participants and the state project liaisons 
prior to, during, and after the field test. It is anticipated that members of the study team will have 
direct responsibility for certain states: 
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• John Bertot/Colleen Ostiguy - Maryland, North Carolina, and Pennsylvania; and 

• Joe Ryan/Chuck McClure - Delaware, Michigan, and Utah. 

This should facilitate communication and issue resolution throughout the field test period. 

CONCLUDING COMMENTS 

It is important to note that this is the first field test of its kind for collecting public library 
network-based statistics. Thus, there will undoubtedly be unforeseen issues encountered during 
the field test period. It is the hope of the study team that the tailored approach to each 
participating state will minimize problems during the field test period in terms of data collection. 
It is not possible, however, to predict with precision what will happen during the field test. 

It is important to maintain a focus on the purpose of the field test: to test the network 
statistics - definitions, operationalization, data collection process, level of effort required, 
reporting of the data, and analysis of the data. Feedback from the field test participants on these 
aspects of the network statistics will provide all of us with extremely useful information as to the 
types of network statistics to collect and the issues involved with their collection. 

The study team appreciates the willingness of the study participants to test the statistics and 
engage in experimentation throughout the data collection process. 
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APPENDIX D.l - FSCS DATA ELEMENT DEFINITIONS 
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Data Element 38 - Reference Transactions 

A reference transaction is an information contact which involves the knowledge, use, 
recommendations, interpretation, or instruction in the use of one or more information sources by 
a member of the library staff. It includes information and referral services. Information sources 
include printed and non-printed materials, machine-readable databases, catalogs, and other 
holdings records, and, through communication or referral, other libraries and institutions and 
people inside and outside the library. The request may come in person, by phone, by fax, mail, 
or by electronic format from an adult, a young adult, or a child. 



Data Element 44 - Operating Expenditures for Library Materials in Electronic Format 

Report operating expenditures for materials considered part of the collection, whether purchased 
or leased, such as CD-ROMs, magnetic tapes, and magnetic discs, that are designed to be 
processed by a computer or similar machine. Examples are U.S. Census data tapes, locally- 
mounted databases, serials, and reference tools. Include operating expenditures for equipment 
when the cost is inseparably bundled into the price of the information service product. Exclude 
operating expenditures for library system software and microcomputer software used only by the 
library staff. 



Data Element 51 - Number of Internet Terminals Used by General Public 

Number of computer terminals (PC, ‘dumb terminal’, etc.) used by general public in the library 
that are used to connect to the Internet (text only, graphical, etc.). 
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STEP-BY-STEP FIELD TEST PROCEDURES 

Thank you for agreeing to participate in the network measures field test. This document 
covers what the study team would like you to do. First some basics: 

• Purpose: Earlier work by the study team, with your help, identified a group of 
network measures that may be useful to public libraries. During the field test we 
would like to develop unambiguous definitions for these measures, clear and 
standardized collection procedures, and identify potential uses of the resulting data. 
The study team will compile this information into a field manual of network 
measures. 

• What each library will do: Each library participating in the field test has been 
assigned a small subset of the network measures to test. The library will receive 
(below) test definitions and data collection procedures for each measure assigned. 
The local library will implement these procedures documenting the step-by-step 
process used, how long it took, and problems and issues that may result. Data should 
be collected for a two week period. The library will also consider ways that it might 
use the data collected. The library will then submit the data collected and a summary 
of procedures used, problems and issues uncovered (if any), and potential uses for the 
data collected to the project’s web site < http://www.albany.edu/~imlsstat/ 
forms/fieldtest.html > by February 1, 2000. 

• Field Test Dates: The library may collect data between January 1, 2000 and January 
31, 2000. We would like you to collect data for a minimum of two weeks. The report 
on your experience is to be submitted to the projects web site by February 1, 2000. 

• Key Contact: There are always unforeseen problems, large and small, that crop up in 
any field test. When they do, please contact the project coordinator, Joe Ryan 
<jryan@mailbox.syr.edu> (315) 475-3630. 

The object of the field test is to discover the potential problems and issues associated with these 
network measures along with the procedures and solutions so that the general public library 
community will have access to the best possible, pre-tested, network measures. The study team 
will use the results of the field test when preparing a field manual of network measures. Your 
library’s input into this process is essential to the study team. So once again, thank you for 
participating. The next section offers some general instructions for participating in the field test. 
This section is followed by the specific data elements the study team would like your library to 
collect including definitions and procedures. 
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General Instructions 

1) Pick a project liaison/coordinator : It may be useful to designate someone on staff to 
coordinate local data collection, interact where necessary with Joe Ryan the project coordinator, 
collect the data, prepare associated reporting and commentary, and input the results to the 
project’s web site. 

2) View the process with a constructively critical eye : Keep the big picture in mind: what would 
you as the local librarian in charge of collecting this data want to know in advance so that the 
data are collected efficiently and used effectively. Please don’t hesitate, or wait for permission, 
to tell the study team anything that will improve the manual they write to accompany these 
measures. 

3) Determine the data collection test period : Most of the measures being tested require that you 
collect data over a two-week period of time. Note that you can collect data any time January 1, 
2000 and January 31, 2000. How you determine the collection period and why will be of interest 
to the study team. The field manual may- well recommend that certain measures be collected 
using sample periods rather than over an entire year. What advice would you offer a library 
when selecting a sample period for any specific measure? 

4) Develop local data collection procedures : Identify the local, step-by-step, procedures 
necessary to collect the network measures assigned. Who is going to do what, when, why? 
Prepare a written, step-by-step plan for collecting this data element, follow the format: 

Data element 

Task Staff Member (Identify by job title) Hours (in quarter hour increments) 

Add the time taken on each task as it is completed. Report the procedures to the study team 
using the project web site. The study team is also interested in obtaining any work forms or 
written documents produced in connection with the data collection process. 

5) Start the clock early : Everyone is concerned with the burden placed on the local libraries that 
will collect these network measures. One way of assessing the burden is by measuring how long 
it takes to complete the measurement task. How long does it take, who, to do what? The study 
team is interested in your planning time, as well as the time needed to collect each measure, and 
time needed to report the results. 

6) Collect the data : The study team is interested in the results as well as the process of collecting 
the data. 

7) Report problems and issues : The study team anticipates that when you start planning to collect 
the data, ambiguities in the definition of the data elements and other problems and issues will 
surface. Other problems and issues with each data element may surface when you actually start 
to collect the data. The study team is interested in what you do, why, any “judgment calls” or 
“what does that dumb study team mean by this” experiences you face, and what you decide to do 
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- no matter how silly or trivial they seem in hindsight. Please note and report these problems and 
issues to the project web site as well 

8) How useful is each data element to your library : The study team is interested in how you rank 
the utility of each data element you collect. How would you use each data element (and why)? 
What do you think of the accuracy or credibility of each data element (having some experience 
collecting them)? What other data would be helpful (e.g., the results from the same data element 
from peer libraries locally or nationally)? How might the data element be used in combination 
with other data elements (e.g., circulation figures are commonly combined with some measure of 
population served to obtain say circulation per capita)? Use the following format: 

Data element: Rank: (use number from l=not at all useful to 5=extremely useful) 

Comments: 

9) Report data to project web site at http://www.albany.edu/~imlsstat/forms/fieldtest.html You 
are welcome to view the web site at any time. The site is designed to collect two types of data 
from your library: 1) The results of your data collection, and 2) A summary of the process and 
procedures used, the time it took, any work forms or written material you developed, problems 
and issues uncovered, your input on the possible uses for each measure, and any general 
comments you think will be helpful to the study team. Instructions for how to input the data are 
provided. 
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What Data Elements Should the Library Collect 

The study team would like you to collect data for the following data elements. Each data 
element has a proposed definition and draft procedures. 

Public Access Workstations 

# Public access workstations 



Definition: Count of the number of public access graphical workstations that connect to the 
Internet. 

Procedure: 

1) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

2) Count the number of graphical workstations with Internet access (no matter the speed or type 
of connection) that are made available to the public as of the end of the field test period. 

In addition to personal computers, workstations may include “thin clients,” graphical terminals, 
or networked computers if they are connected to the Internet and publicly accessible. Computers 
in computer labs used for public instruction if graphical and connected to the Internet should be 
counted. Public access graphical workstations that connect to the Internet that are used by both 
staff and the public should be counted if the workstation is used by the public for at least half of 
the hours during an average week that the library is open to the public. Reference desk 
computers used by staff to assist the public should not be counted. 

3) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

3) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

4) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

5) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

# Public access workstation users 



Definition: Count of the number of users of all of the library’s graphical public access 
workstations connected to the Internet during a week [in the case of the field test]. 
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Procedure: 

1) Select a two-week period during the test period. One week equals the number of hours the 
library is open over a consecutive seven-day period. 

2) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. Identify the dates of the weeks chosen in the written plan. The 
number of users may be counted by observation (continuous or every X minutes), manually 
using registration sign up sheets, or via computer software such as Historian. You decide which 
method to use (and report your choice in the written procedure plan above). Indicate in your 
plan whether you observed (and for how long), used sign up sheets, or used software to collect 
the data for this element. If you used software please indicate the name, approximate cost, 
address, phone and web page of the software used. 

3) Count the number of users of all of the library’s graphical public access workstations 
connected to the Internet during each sample week chosen. 

Count each user that uses the graphical public access workstations connected to the Internet, 
regardless of the amount of time spent on the computer. A user, who uses the library’s 
workstations three times a week, would count as three users in the count. Internet use includes all 
types of usage including WWW, e-mail, telnet, chat, etc. The study team recognizes the potential 
difficulty of determining whether a user on a multi-purpose (cd-rom access, word processing, 
etc.) workstation is using the Internet. Do not include staff use of these resources. 

4) Obtain an average weekly use figure by adding the total number of users each week and 
dividing by the number of weeks this data element was surveyed. For example during the first 
week 70 users were counted, 80 users were counted the second week. An average of 75 users is 
obtained and reported by adding week l’s users to week 2’s [70 + 80 = 150] and dividing by the 
number of weeks surveyed [150 h- 2 = 75], 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

Maximum speed of public access Internet workstations 

Definition: Indication of the maximum speed of public Internet access, e.g., 56kbps, ISDN, Tl. 
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Procedure: 

1) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

2) Determine what is the maximum speed of Internet access offered by the library to your users 
at your public access workstations as of the end of the field test period. 

The number requested is not an average. This number is the maximum speed offered even if it is 
only at one public access workstation. The speed of Internet access offered to staff does not 
count here. Common measures of speed include: 56kbps, ISDN, Tl. If you only offer Internet 
access by modem, look on the modem or in its manual for an indication of the maximum speed. 
If you offer Internet access via an Internet Service Provider (ISP) this is likely to be the source of 
your maximum speed connection. Contact your ISP to obtain the maximum speed you have 
purchased (or otherwise obtained). 

3) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

4) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

5) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

6) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 



Electronic Services 



# Electronic reference transactions 



Draft Definition: Count of the number of reference questions received electronically (e.g., via e- 
mail, WWW form, etc.) per week [in the case of the field test]. 

Draft Procedure: 

1) Select a two-week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

2) Prepare a written, step-by-step plan for collecting this data element, following the format 
outlined under General Instructions. 

3) Count the number of electronic reference requests received during the each week of the 
sample period. Requests may be sent via e-mail, fax, a form on a web page, etc. Report an 
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electronic reference transaction as you would a face-to-face reference transaction. Thus, for 
example, one e-mail request may contain several reference questions taking varying time to 
complete. For example, one e-mail request contained 1 ready-reference question and one 
reference question that took 10-15 minutes to answer. Count the number of questions not the 
number of requests. So in the example you would report 2 as the number of electronic reference 
transactions. 

4) Obtain an average weekly number of electronic reference requests received figure by adding 

the total number of electronic reference requests received each week and dividing by the number 
of weeks this data element was surveyed. For example during the first week 7 electronic 
reference requests were counted, 9 were counted the second week. An average of 8 electronic 
reference requests is obtained and reported by adding week l’s users to week 2’s [7 + 9 = 16] 
and dividing by the number of weeks surveyed [16 2 = 8]. 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/fonns/fieldtest.html>. 

Virtual Visits 

# Virtual visits to networked library resources excluding in-library use 



Definition: A visit occurs when an external user connects to a networked library resource for any 
length of time or purpose (regardless of the number of pages or elements viewed). This may be a 
library OP AC or a library web page. In the case of a user visit to a library web site a user who 
looks at 16 pages and 54 graphic images registers one visit on the Web server. Due to various 
web server issues and differing software this measure is an estimate of the visits to the web site. 
One definition (from the MS Site Server manual) of a virtual visit is: “A series of consecutive 
requests from a user to an Internet site. If your log file data includes referrer data, then new 
visits begin with referring links external to your Internet site. Regardless of whether or not you 
have referrer data, if a user does not make a request after a specified time period, the previous 
series of requests is considered to be a completed visit.” 

Procedure: 

1) The library must offer external electronic access to one or more of its resources. For example, 
a library web page, library OP AC, networked CD-ROM databases that can be accessed by 
library users located external to the library. If you do not offer such a service do not report on 
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this data element. Instead please contact the local coordinator of this field test or Joe Ryan 
<jryan@mailbox.syr.edu> if you have a question. 

2) Select a two-week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

3) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. Indicate the software used to collect the virtual visit data. Identify 
the software data element chosen to collect virtual visits and (where possible) provide the 
software’s definition for this element (often found in the software manual). In the field manual 
the study team will produce, we would like to include appropriate data elements and definitions 
from appropriate software. Your help here will make that possible. Report the method used and 
your success in excluding in library and staff use of your electronic services. 

4) Count the number of virtual visits each week during the two-week period chosen. Exclude 
(where possible) virtual visits by staff and by library users using public access workstations in 
the library . This exclusion is made because the virtual visit count may be combined with the 
turnstile count of visitors to the library’s physical premises to obtain a more representative count 
of visitors to the library. So the virtual visits from those using equipment within the library are 
excluded to reduce double counting. 

Include (where possible) a count of the number of remote logins (sessions) to non-web-based 
library resources such as OP ACS as well. This may require the use of several different software 
packages to achieve. 

Configure software and local technology to collect virtual visit data. The virtual visit count is 
obtained using computer software often called log analysis software. All log analysis software 
may not track virtual visits the same way so the count obtained will necessarily be an estimate. 
Additional changes in local configuration may be required to ensure the same data is collected. 

5) Obtain an average weekly number of virtual visits figure by adding the total number of virtual 
visits each week and dividing by the number of weeks this data element was surveyed. For 
example, during the first week 100 virtual visits were counted, 150 were counted the second 
week. An average of 125 virtual visits is obtained and reported by adding week l’s virtual visits 
to week 2’s [100 + 150 = 250] and dividing by the number of weeks surveyed [250 + 2 = 125], 

6) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

7) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

8) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 
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9) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

Training 



# Users trained 



Definition: Count of the number of users trained in structured and informal training sessions 
conducted by the library. Structured means a course with a designed curriculum intended to 
demonstrate the use of a technology such as the Web, Internet searching, personal computing, 
etc. Informal includes contacts with users by library staff intended to demonstrate the use of 
library workstations, aspects of the applications available on those workstations, etc. 

Procedure: 

1) Select a two-week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

2) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

3) Count the number of users trained each week during the two-week period chosen. Include a 
count of all users attending all formal, structured courses the library offers, the library contracts 
for, or that use library facilities. Include informal training of users at various public service 
locations in the library (e.g., the reference desk) whose duration is fifteen minutes or more. 
Include only training in the use of information technology or resources obtainable using 
information technology. Examples include use of the WWW, Internet searching, use of public 
access workstations or personal computers, subject-based resources available on the Internet, 
social implications of information technology (e.g., filtering and the public library). 

Exclude staff training here. Exclude ready reference and traditional reference questions using 
pre-existing, locally developed guidelines. Exclude user training not in information technology 
related areas (examples above). 

A user need not be a registered library user. A single individual may attend multiple training 
sessions of the same or different types, each of which is counted. So if a single individual 
attended a structured introduction to the Internet session and received a fifteen minute informal 
introduction to employment resources on the Internet at the reference desk the # users training 
count would increase by 2. 

4) Obtain an average weekly number of users trained figure by adding the total number of users 
trained each week and dividing by the number of weeks this data element was surveyed. For 
example, during the first week 50 users receiving training were counted, 100 were counted the 
second week. An average of 75 users trained is obtained and reported by adding week l’s users 
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trained to week 2’s [50 + 100 = 150] and dividing by the number of weeks surveyed [150 4- 2 = 
75], 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

# Hours of user training 



Definition: Count of the number of hours library staff spends training users in structured and 
informal training sessions conducted by the library. Structured means a course with a designed 
curriculum intended to demonstrate the use of a technology such as the Web, Internet searching, 
personal computing, etc. Informal includes contacts with users by library staff intended to 
demonstrate the use of library workstations, aspects of the applications available on those 
workstations, etc. 

Procedure: 

1) Select a two-week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

2) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

3) Count the number of hours of user training each week during the two-week period chosen. 
Include all formal, structured courses the library offers, the library contracts for, or that use 
library facilities. Include informal training of users at various public service locations in the 
library (e.g., the reference desk) whose duration is fifteen minutes or more. Include only training 
in the use of information technology or resources obtainable using information technology. 
Examples include use of the WWW, Internet searching, use of public access workstations or 
personal computers, subject-based resources available on the Internet, social implications of 
information technology (e.g., filtering and the public library). 

Exclude staff training here. Exclude ready reference and traditional reference questions using 
pre-existing, locally developed guidelines. Exclude user training not in information technology 
related areas (examples above). Exclude staff preparation time. 
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Report hours in quarter hour increments. For example a user training class lasting 70 minutes 
should be counted as 1.25 hours. 

4) Obtain an average weekly number of hours of user training figure by adding the total number 
of hours of user training each week and dividing by the number of weeks this data element was 
surveyed. For example, during the first week 10 hours of user training were counted, 20 were 
counted the second week. An average of 15 hours of user training is obtained and reported by 
adding week 1 ’s hours of user training to week 2’s [10 + 20 = 30] and dividing by the number of 
weeks surveyed [30 + 2 = 15], 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

# Staff trained 



Definition: The total number of staff trained through formal/structured information technology 
training during a sample period. Structured means a course with a designed curriculum intended 
to demonstrate the use of a technology such as the Web, Internet searching, personal computing, 
etc. 

Procedure: 

1) Select a two-week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

2) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

3) Count the number of staff trained each week during the two-week period chosen. Include all 
formal, structured courses the library offers, the library contracts for, or training sources external 
to the library (e.g., college and university courses, professional development courses at 
conferences). Include only training in the use of information technology or resources obtainable 
using information technology. Examples include use of the WWW, Internet searching, use of 
public access workstations or personal computers, subject-based resources available on the 
Internet, social implications of information technology (e.g., filtering and the public library). 
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Exclude informal staff training. Exclude training not in information technology related areas 
(examples above). 

A single staff member may attend multiple training sessions of the same or different types, each 
of which is counted. So if a single staff member attended a course on using the Internet at a local 
community college and attended a workshop on Internet resources on aging the # staff trained 
count would increase by 2. 

Staff include professional, non-professional, and volunteer staff as well as board members. 

4) Obtain an average weekly number of staff trained figure by adding the total number of staff 
trained each week and dividing by the number of weeks this data element was surveyed. For 
example, during the first week 20 staff receiving training were counted, 30 were counted the 
second week. An average of 25 staff trained is obtained and reported by adding week l’s staff 
trained to week 2’s [20 + 30 = 50] and dividing by the number of weeks surveyed [50 + 2 = 25], 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 

# Hours of staff training 



Definition: Count of the number of total hours staff received formal/structured technology 
training. 

Procedure: 

1) Select a two- week period during the test period. The study team may propose that this 
measure be collected for the entire year in the final report rather than two weeks. Identify the 
dates of the weeks chosen in the written plan. 

2) Prepare a written, step-by-step plan for collecting this data element, follow the format outlined 
under General Instructions. 

3) Count the number of hours of staff training each week during the two-week period chosen. 

Include all formal, structured courses the library offers, the library contracts for, or training 
sources external to the library (e.g., college and university courses, professional development 
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courses at conferences). Include only training in the use of information technology or resources 
obtainable using information technology. Examples include use of the WWW, Internet 
searching, use of public access workstations or personal computers, subject-based resources 
available on the Internet, social implications of information technology' (e.g., filtering and the 
public library). 

Exclude informal staff training. Exclude training not in information technology related areas 
(examples above). 

A single staff member may attend multiple training sessions of the same or different types, each 
of which is counted. So if a single staff member attended a course on using the Internet at a local 
community college lasting 6 hours and attended a workshop on Internet resources on aging 
lasting 1 hour the # Hours of staff training count would increase by .7 hours 

Staff include professional, non-professional, and volunteer staff as well as board members. 

Report hours in quarter hour increments. For example a training class lasting 70 minutes should 
be counted as 1.25 hours. 

4) Obtain an average weekly number of hours of staff training figure by adding the total number 
of hours of staff training each week and dividing by the number of weeks this data element was 
surveyed. For example, during the first week 10 hours of staff training were counted, 20 were 
counted the second week. An average of 15 hours of staff training is obtained and reported by 
adding week l’s hours of staff training to week 2’s [10 + 20 = 30] and dividing by the number of 
weeks surveyed [30 4- 2 = 15]. 

5) Be sure to identify time-on-task data using the format discussed under the General 
Instructions. 

6) Identify any problems and issues that develop when collecting this data element and report 
them on the project web page. 

7) Assess and rank the utility of this network measure and report this to the web site using the 
format suggested under General Instructions. 

8) Report this information following the instructions provided to the project web site 
<http://www.albany.edu/~imlsstat/forms/fieldtest.html >. 
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APPENDIX F - ONLINE DATABASE VENDOR FIELD TEST METHODOLOGY 
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INTRODUCTION 

This document serves as a both an outline and instruction manual for field-testing the database data 
elements/statistics. North Carolina through NC LIVE, Maryland through Sailor Operations Center 
(SOC), and Delaware through DelAWARE will be field-testing the database data elements/statistics. The 
study team developed the document to better aid the study participants (Maryland, North Carolina, 
Delaware, and licensed database vendors) in the field-testing phase of the project. The documentation 
includes: 

• Information regarding what is expected of both the study participants and licensed database 
vendors; 

• Detailed definitions of the database data elements/statistics; 

• Preferred method(s) of collecting and reporting the requested information; 

• Study team contact information; and 

• Internet links to online help and comment forms developed by the study team. 

Together, these items comprise the database field test component of the study. 

EXPECTATIONS 



Study Participants 

The study team intends to work with Maryland, North Carolina, and Delaware to collect the database 
usage statistics. We anticipate that the study participants will do the following: 

• Establish a contact person: 

0 Maryland - at Gale/IAC through the state library and Sailor Operations Center (SOC) or 

Division of Library Development and Services (DLDS) management structures; 

0 North Carolina - at Ebsco, OCLC, and UMI through NC LIVE and state library 

management structures to explain the nature of the project, the desired data elements, data 
collection activities, and other relevant information; and, 

0 Delaware - at Ebsco and WorldBook through the DelAWARE and state library 

management structures. 

• Work collaboratively with the study team and vendor staff to get database usage reports for 
analysis purposes in a standardized electronic file format; 

• Test the accuracy, method, and format of the data reported; 

• Determine the types of breakdowns desired/most useful to the database licensee; 

• Discuss, for each data element, any problems or issues that had to be resolved during data 
collection and how the issues were resolved; and 

• Suggest how the data collection effort for each data element might be improved in the future. 

The above will assist the study team, participants, and vendors identify key issues concerning the 
availability, collection, analysis, and dissemination of online database vendor statistics. 
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Vendors 

The study team is asking each licensed database vendor to attempt to collect four database statistics: 

1 . Number of unique electronic titles; 

2. Number of electronic network sessions; 

3. Number of electronic network queries/searches; and, 

4. Number of electronic networked items examined. 

As such, the study team expects each licensed database vendor to do the following: 

• Work collaboratively with study team, the project coordinators in Maryland and North Carolina, 
and the management teams for the networked resources in each state; 

• Communicate with the study team to identify and resolve any key data collection, definitional, or 
other issues; 

• Collect the statistics as defined by the study team (and ICOLC); 

• Provide a detailed, step by step description of the process employed to collect the statistics 
including any forms developed to collect them; 

• Include as many statistical refinements where applicable; and 

• Suggest how the data collection effort for each data element might be improved in the future. 

This approach will provide the study team with useful information regarding the vendor issues in 
providing a variety of database usage data in a number of formats. 

DATA COLLECTION 

This section will provide the study participants in Maryland and North Carolina and the licensed database 
vendors the following: 

• Study team contact information; 

• Links to online help and comment forms; and 

• Instructions on how to collect the database data elements/statistics. 

Study Team Contact Information 

The study team is aware that questions and issues will arise during field-testing. Thus, it is necessary 
that the study participants know whom to contact. The following two study team members to contact are: 



Name 


Phone Number 


Fax Number 


Email Address 


John Bertot 


(518) 442-5125 


(518) 442-5367 


<jcbertot@cnsunix.albany.edu> 


Colleen Ostiguy 


(518) 442-5124 


(518) 442-5367 


<imlsstat@cnsunix.albany.edu> 
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Online Forms 

The study team created online comment and help forms. When a study participant submits 
an online form it is: 

• Sent immediately via email to a study team member; and 

• Depending upon the question/comment, answered via email or telephone. 

The online help and comment forms are located at: <http://www.albany.edu/-imlsstat/database.html> 

Field Test Instructions 

The following are instructions for the collection and reporting of the five database data 
elements/statistics. 



Number of Unique Electronic Titles 

Definition : Count the number of unique online database titles available at the library. 

Participant Responsible for Collection : Licensed Database Vendor(s) 

Instructions for Collection : 

1. Count and identify the number of online database titles available from the entire host/service to which 
the licensee subscribes. 

2. Count and identify the number of full text online database titles available. 

3. Count and identify the number of abstracted online database titles available. 

4. Report statistics to study team in electronic form. 

5. Email statistics to <imlsstat@cnsunix.albany.edu> 



Number of Electronic Network Sessions 

Definition : Count of the number of initial sessions (logins) to the online database. 

Participant Responsible for Collection : Licensed Database Vendor(s) 

Instructions for Collection : 

1 . Count the number of logins to entire host to which the licensee subscribes for month of January 
2000 . 

2. Breakdown number of logins to entire host by time of day and IP address. 

3. Count the number of logins to each database for month of January 2000. 

4. Breakdown number of logins to each database by time of day and IP address. 

5. Report statistics to study team in electronic form, e.g. ASCII delimited file or downloadable file. 

6. Email statistics/file location to <imlsstat@cnsunix.albany.edu> 
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Number of Electronic Network Queries/Searches 

Definition : Count the number of searches conducted in the online database. 

Participant Responsible for Collection : Licensed Database Vendor(s) 

Instructions for Collection : 

1. Count the number of searches conducted in entire host to which the licensee subscribes for month of 
January 2000. 

2. Breakdown number of searches conducted in entire host by time of day and IP address. 

3. Count the number of searches conducted in each database for month of January 2000. 

4. Breakdown number of searches conducted in each database by time of day and IP address. 

5. Provide the search strings/queries conducted in each database for month of January 2000. 

6. Report statistics to study team in electronic form, e.g. ASCII delimited file or downloadable file. 

7. Email statistics/file location to <imlsstat@cnsunix.albany.edu> 



Number of Electronic Network Items Examined 

Definition : Count the number of online database content views. View defined as a user looking at an 
abstract or full text article 

Participant Responsible for Collection : Licensed Database Vendor(s) 

Instructions for Collection : 

1 . Count the number of content views of abstract or full text article(s) in entire host to which the 
licensee subscribes for month of January 2000. 

2. Breakdown number of content views to entire host by: 

o User action (printing, emailing, or saving to disk); and 
o Type of documents (PDF, text, image, or video). 

3. Count the number of content views of abstract or full text article(s) in each database for month of 
January 2000. 

4. Breakdown number of content views in each database by: 
o User action (printing, emailing, or saving to disk); and 
o Type of documents (PDF, text, image, or video). 

5. Report statistics to study team in electronic form, e.g. ASCII delimited file or downloadable file. 

6. Email statistics/file location to <imlsstat@cnsunix.albany.edu> 
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Internet Subscription Material Expenditures 

Definition : Total expenditures for online Internet material subscription expenditures (vendor databases). 
May include one-time start-up cost as well as recurring costs. 

Participant Responsible for Collection : Study participants in Maryland, North Carolina, and DelAWARE. 
Instructions for Collection : 

1 . Calculate the one-time start-up fee(s) for each licensed database(s). 

2. Calculate the recurring cost(s)/fee(s) for each licensed database(s). 

3. Itemize each fee so study team can understand how final figures were configured. 

4. Report information to study team. 

5. Email statistics to <imlsstat@cnsunix.albany.edu> 



EMAIL FOLLOW-UP TO VENDORS 

During this week I will be contacting a few vendors to request their participation with the 
"database usage statistics" segment of the project. And, we have secured participation from three 
statewide networks: Maryland's Sailor Public Online Information Network, North Carolina's 
NCLIVE, and Delaware’s DelAWARE. From contacts at each of these state networks we 
understand that Gale has agreements with Maryland and North Carolina. 

We are requesting that you work with us in identifying what statistics are currently delivered to 
these customers and if the monthly reports are available in an ASCII comma delimited format. 
And, for the purposes of standardizing what database use statistics are collected and reported to 
customers, we are asking that you participate in an alpha test of reporting in a standardized 
format and sequence. The alpha test period is December 1990-February 2000. 

Let me begin with the standard data elements in the four (4) identified major categories of data 
collection: 

1. Number of unique electronic titles: count the number of unique titles in online databases 
available in the library; 

2. Number of electronic network sessions: count the number of initial sessions (logins) to the 
online database; 

3. Number of electronic network queries: count the number of searches conducted in the online 
database; and 

4. Number of electronic network items examined: count the number of online database content 
views. View defined as a user looking at an abstract or full text article. 

Within each category there is more information requested. I have attached a Word document 
with further description of each data category. 

As far as standardized format and sequence of usage reports, we are asking participating vendors 
to report as follows: 
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1 . ASCII text, comma delimited files 

2. usage reports include the following information: 

a. customer label 

b. IP address for each library in license (we are trying to collect data by library and IP address, 
so be as specific as you can with this item. Ideally we want usage reports by each IP address in a 
library license) 

c. total sessions 

d. total rejected sessions (tum-aways) 

e. databases searched 

f. number of searches by subject, providing top 20 subjects searched in each database) 

g. queries by database examined - citation, abstract, full text, type of document (ASCII text, pdf, 
color image, sound clip, etc.) 

I cannot tell you how important this project is both for customers and for vendors. If we can in 
any way influence the process for standardizing data use reports requested of vendors this project 
has done its job. And, it standardizes what libraries are requested to report in state and national 
library surveys. Clearly, we create a win-win situation for all concerned. 

Our timeline for knowing the extent to which you can participate and provide the level of use 
data in the format we request is December 10, 1999. Since XXX is such a significant presence in 
the library database market and with our test libraries, your participation is appreciated. And, 
XXX participation only underscores the company’s commitment to improving services to 
customers and the users they serve. 
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